vSphere CSI Volume Expansion Fails with CnsFault VSLM Task Failed and Stale ClaimRef

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• In VMware Tanzu Kubernetes Grid (TKG) environments utilizing the vSphere CSI driver, attempts to expand a PersistentVolumeClaim (PVC) or rebind a PersistentVolume (PV) may fail.

Environment

2.5.4

Cause

The issue is caused by a stale claimRef within the PersistentVolume metadata. When a PVC is deleted, the corresponding PV with a Retain reclaim policy enters a Released state. However, the spec.claimRef field continues to hold the UUID of the deleted PVC. This prevents the Kubernetes PV controller and the vSphere CSI resizer from performing operations (like expansion or rebinding) because the volume is still technically claimed by a non-existent resource.

Resolution

To resolve this issue, the stale claimRef must be manually cleared to transition the PV back to an Available state.

1. Scale the deployment or statefulset to zero to ensure no active I/O is targeting the volume.
   kubectl scale sts <statefulset_name> -n <namespace> --replicas=0
       
2. Identify the PV associated with the failed PVC and check its status.
   kubectl get pv <pv_name>
       
3. Use a merge patch to set the claimRef to null. This will transition the PV status from Released to Available.
   kubectl patch pv <pv_name> --type=merge -p '{"spec":{"claimRef":null}}'
       
4. Recreate or apply the PVC:
   Apply the PVC configuration. Ensure the volumeName field in the PVC spec matches the <pv_name>.
   kubectl apply -f <pvc_definition>.yaml
       
5. Confirm the PVC status has changed to Bound.
   kubectl get pvc <pvc_name> -n <namespace>
       
6. Once the volume is bound, scale the workload back to the original replica count.
   kubectl scale sts <statefulset_name> -n <namespace> --replicas=<count>