"exceeded maximum number of permitted snapshots" error persists after snapshots have been removed from Persistent Volume Claims

Products

• VMware vCenter Server

Issue/Introduction

After removing PVC snapshots, attempts to snapshot the volume fail with the error "exceeded maximum number of permitted snapshots". 
This error normally triggers when reaching the maximum number of snapshots per volume.  
*See the following documentation for guidance on Volume Snapshot and Restore

However, this error can trigger when the number of PVC snapshots is below the maximum number permitted if stale snapshot information persists within the CNS database tables. 

Environment

VMware vSphere Container Storage Plug-in 2.0
VMware vSphere Container Storage Plug-in 3.0

VMware vCenter Server Appliance 7.0

NOTE: This issue is NOT found in VMware vCenter Server Appliance 8.0

Cause

This issue occurs when the ESXi hostd agent removes the PVC snapshot successfully, but external issues prevent the vCenter vpxd service from recognizing task completed, leading to stale entries within the CNS snapshot database table. 

Resolution

Validate the Number of Snapshots Present for the PVC

- If not configured granularly, the default value for maximum snapshots is 3. 
- If configured granularly, the maximum number of snapshots will be determined in the snapshot section of /etc/kubernetes/csi-vsphere.con

Run the following command: kubectl get volumesnapshots
Review the MOB to validate the number of snapshots present for each volume ID
- Navigate to https://vc_fqdn/mob/?moid=VStorageObjectManager&method=retrieveSnapshotInfo
 

The values from these commands should be equal and at or below the maximum snapshot limit

If they are below the snapshot limit and the "exceeded maximum number of permitted snapshots" error persists when attempting to snapshot, please engage Broadcom support for further troubleshooting and remediation.