CSI Migration stuck in progress for volumes

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)

Issue/Introduction

This article provides an insight on vSphere Container Storage Plug-in migration process referred to in the VMware CSI documentation

Symptoms:
After upgrading TKGi from 1.13 to 1.14 , CSI Migration could be stuck in progress.

• Volumes could be having a failed migration process with the below error

Warning FailedAttachVolume 40m (x9 over 43m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-<ID>" : rpc error: code = Internal desc = failed to get VolumeID from volumeMigrationService for volumePath: "[<ID>] kubevols/<ID>.vmdk"

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

VMware vSphere Cloud Native Storage (CNS)

VMware vSphere Container Storage Interface (CSI) Driver 2.x

Cause

PV and PVC yaml files are missing annotations that caused the migration to be failed on some volumes.

Resolution

As per this VMware Tech Docs page, statically provisioned in-tree volumes must have specific annotations.

Adding the missing annotations manually to the PVs and PVCs yaml file and then deleting the pods associated will solve the issue.

Annotation on PV:

annotations: 
   pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume

 

Annotation on PVC:
 

annotations:
  volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume

 

 

Additional Information

Impact/Risks:
CSI Migration stuck in progress for volumes