Persistent Volume fails to expand/resize with "Please delete snapshots before expanding the volume" error

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

• Expanding/resizing a PVC/PV (PersistentVolumeClaim/PersistentVolume) fails with an event error:
  
  I0205 12:42:53.362069 161459 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"***", UID:"***", APIVersion:"v1", ResourceVersion:"***", FieldPath:""}): type: 'Warning' reason: 'VolumeResizeFailed' resize volume "pvc-***" by resizer "csi.vsphere.vmware.com" failed: rpc error: code = FailedPrecondition desc = volume: *** with existing snapshots [size_bytes:17179869184 snapshot_id:"***" source_volume_id:"***" creation_time:<seconds:1737939804 nanos:449999000 > ready_to_use:true size_bytes:17179869184 snapshot_id:"***" source_volume_id:"***" creation_time:<seconds:1737939804 nanos:449999000 > ready_to_use:true ] cannot be expanded. Please delete snapshots before expanding the volume
  
  
• KB Failed to expand volume because the disk that backs it has snapshots shows how to check and delete snapshots for a particular volume.
  If "govc disk.snapshot.ls -k -dc=<datacenter-name> -ds=<datastore-name> -l <volume-id>" doesn't show any snapshot for the <volume-id>, check the Resolution section below.

Environment

TKGi and vCenter 7.0.3

Cause

It is believed the root cause of the issue is a limitation in VM-based serialization in vCenter version 7.0u3. This version allows multiple operations, such as deleteSnapshot and createDiskFromSnapshot, to be performed concurrently on different disks attached to the same VM. This concurrency can lead to conflicts where the snapshot deletion fails because a CreateDiskFromSnapshot operation is running on the same snapshotted disk. As a result, the snapshot ID entry is removed from the disk before consolidation, leading to space leakage

Resolution

Most likely, stale snapshot entries associated to the <volume-id> are present in the VCDB CNS table and need to be removed.

Please, open a Support Request with the CNS team to get guidance on the resolution:

Support Product: VMware vCenter Server

Component Name: Cloud Native Storage