Warning VolumeResizeFailed - not in FileSystemResizePending condition

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

When using VKS a PVC is unable to be resized. Running a describe on the PVC shows the message:

Warning  VolumeResizeFailed  3m35s (x171 over 12h)  external-resizer csi.vsphere.vmware.com  resize volume "pvc-<uuid>" by resizer "csi.vsphere.vmware.com" failed: rpc error: code = Internal desc = failed to expand volume <volume handle> in namespace <namespace> of supervisor cluster. Error: supervisor PersistentVolumeClaim <volume-handle> in namespace <namespace> not in FileSystemResizePending condition and XXGi size within XX seconds.

Environment

VKS with vSphere 8.0+

Cause

A snapshot volume may exist on the persistent volume and it cannot be resized.

Resolution

1. Locate the volume handle from the Supervisor context.
   1. From Guest Cluster context run:
      kubectl get pv <PV_WITH_SNAPSHOT> -o jsonpath='{.spec.csi.volumeHandle}{"\n"}'
   2. Next, change to Supervisor context then run:
      kubectl get pvc <Value_From_Step_1> -o jsonpath={'.spec.volumeName}{"\n"}'
   3. Then run:
      kubectl get pv <Value_From_Step_2> -o jsonpath={'.spec.csi.volumeHandle}{"\n"}'
2. Use the volume handle with govc to remove the snapshot volume.
   • Download and install the govc CLI. 
   • From govc:
     
     • List snapshots: 
        govc disk.snapshot.ls -ds <DATASTORE_OF_PV> <Value_From_Step_3>
     • Remove snapshot: 
       govc disk.snapshot.rm -ds <DATASTORE_OF_PV> <Value_From_Step_3> <SNAPSHOT_ID_FROM_disk.snapshot.ls>

Once removed resize will proceed as expected.