Velero backup fails with error "the number of snapshots on the source volume <volume-id> reaches the configured maximum (3)"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

• Velero CSI-based backup fails on TKGi:

# velero backup get

NAME                                 STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR

<backup-name>                        PartiallyFailed   11       474        2025-02-04 01:00:12 +0000 UTC   6d        default            <none>

• velero pod logs show entries as below:

time="2025-02-04T01:00:32Z" level=warning msg="VolumeSnapshotContent snapcontent-<> has error: Failed to check and update snapshot content: failed to take snapshot of the volume <volume-id>: \"rpc error: code = FailedPrecondition desc = the number of snapshots on the source volume <volume-id> reaches the configured maximum (3)\"" backup=velero/<backup-name> cmd=/velero logSource="pkg/util/csi/volume_snapshot.go:744" pluginName=velero

• "kubectl get volumesnapshot,volumesnapshotcontent -A" doesn't show any volume snapshot associated to <volume-id>.
  
  
• On vCenter CLI, "govc disk.snapshot.ls -k -dc=<datacenter-name> -ds=<datastore-name> -l <volume-id>" doesn't show any snapshot for the <volume-id>.
  Instructions on how to install and use govc can be found in KB https://knowledge.broadcom.com/external/article/313099/failed-to-expand-volume-because-the-disk.html 

Environment

TKGi and vCenter 7.0.3

Cause

It is believed the root cause of the issue is a limitation in VM-based serialization in vCenter version 7.0u3. This version allows multiple operations, such as deleteSnapshot and createDiskFromSnapshot, to be performed concurrently on different disks attached to the same VM. This concurrency can lead to conflicts where the snapshot deletion fails because a CreateDiskFromSnapshot operation is running on the same snapshotted disk. As a result, the snapshot ID entry is removed from the disk before consolidation, leading to space leakage

Resolution

Most likely, stale snapshot entries associated to the <volume-id> are present in the VCDB CNS table and need to be removed.

Please, open a Support Request with the CNS team to get guidance on the resolution:

Support Product: VMware vCenter Server

Component Name: Cloud Native Storage