CSI Migration stuck in progress for volumes

Products

• Tanzu Kubernetes Grid

Issue/Introduction

This article provides an insight on vSphere Container Storage Plug-in migration process referring to the below documemtation
https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/2.0/vmware-vsphere-csp-getting-started/GUID-968D421F-D464-4E22-8127-6CB9FF54423F.html

Symptoms:
After upgrading TKGi from 1.13 to 1.14 , CSI Migration could be stuck in progress.

• Volumes could be having a failed migration process with the below error

Warning FailedAttachVolume 40m (x9 over 43m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-<ID>" : rpc error: code = Internal desc = failed to get VolumeID from volumeMigrationService for volumePath: "[<ID>] kubevols/<ID>.vmdk"

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Cause

PV and PVC yaml files are missing annotations that caused the migration to be failed on some volumes.

Resolution

As per the following document https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/2.0/vmware-vsphere-csp-getting-started/GUID-968D421F-D464-4E22-8127-6CB9FF54423F.html, statically provisioned in-tree volumes must have specific annotations.

Adding the missing annotations manually to the PVs and PVCs yaml file and then deleting the pods associated will solve the issue.

Annotation on PV:

annotations: 
   pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume

 

Annotation on PVC:
 

annotations:
  volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume

 

Additional Information

Impact/Risks:
CSI Migration stuck in progress for volumes