CSI Snapshots for PVCs Limited to 4 by Default in VCF 9.1 and 8.0U3i

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Starting with VCF 9.1, the number of Cloud Native Storage (CNS) snapshots for a Persistent Volume Claim (PVC) is restricted to a default limit of four snapshots per volume. If the number of snapshots exceeds this default or a custom configured limit, new snapshot operations will be prevented. Existing snapshots remain unaffected.

"snapshot limit exceeded for PVC ###############. Current snapshots: 9, Limit: 4"

Environment

VMware Cloud Foundation (VCF) 9.1

vCenter Server 8.0 U3i

CNS

VKS

Cause

To optimize performance and storage management, VCF 9.1 introduces a default limit of 4 CSI snapshots per volume. This is enforced via a two-tiered check:

1. PVC Count Check: A simple check of the VolumeSnapshot count for the target PVC.
2. CNS Backend Check: A verification in the CNS backend to ensure the total number of snapshots does not exceed the configured maximum.

Resolution

Users can override the default limit (maximum allowed value is 32) by creating a ConfigMap in the specific namespace.

Create a configuration file named cns-csi-limits.yaml with the following content in the supervisor namespace

cns-csi-limits.yaml 

apiVersion: v1 
kind: ConfigMap 
metadata: 
  name: cns-csi-limits 
  namespace: <your-namespace> 
data: 
  max-snapshots-per-volume: "20"  ===> This example changes the limit to 20.

Apply the ConfigMap to your namespace:

kubectl create -f cns-csi-limits.yaml -n <your-namespace>

Note: The maximum value for "max-snapshots-per-volume" is 32. 

If limits are exceeded, delete existing VolumeSnapshots to allow new snapshots.

kubectl -n <namespace> delete volumesnapshot <volumesnapshot-name>

VolumeSnapshot deletion should be done at the source of the volumesnapshot, i.e., if the volumesnapshot was initiated from the guest cluster then it must be deleted from the guest, and if it's done directly on the supervisor, then delete them in supervisor.

If the issue is observed despite deleting all volumesnapshots, it is likely because of orphan snapshots and need to be cleaned using govc

Additional Information

If the number of snapshots exceeds the default or the configured limit, then no new snapshots will be allowed, but this will not affect existing snapshots.

Users can delete the VolumeSnapshots using kubectl, or also delete the snapshots from cns backed using govc.  Refer to govmomi releases and installation steps.