Velero backup fails with error "the number of snapshots on the source volume <volume-id> reaches the configured maximum (3)"

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Velero CSI-based backup fails on TKGi:

# velero backup get

NAME                                 STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR

<backup-name>                        PartiallyFailed   11       474        YYYY-MM-DD HH:MM:SS +0000 UTC   6d        default            <none>

• velero pod logs show entries as below:

time="YYYY-MM-DDTHH:MM:SSZ" level=warning msg="VolumeSnapshotContent snapcontent-<> has error: Failed to check and update snapshot content: failed to take snapshot of the volume <volume-id>: \"rpc error: code = FailedPrecondition desc = the number of snapshots on the source volume <volume-id> reaches the configured maximum (3)\"" backup=velero/<backup-name> cmd=/velero logSource="pkg/util/csi/volume_snapshot.go:744" pluginName=velero

• "kubectl get volumesnapshot,volumesnapshotcontent -A" doesn't show any volume snapshot associated to <volume-id>.
  
  
• On vCenter CLI, "govc disk.snapshot.ls -k -dc=<datacenter-name> -ds=<datastore-name> -l <volume-id>" doesn't show any snapshot for the <volume-id>.
  
  Instructions on how to install and use GOVC can be found in KB Failed to expand volume because the disk that backs it has snapshots

Environment

CNS and vCenter 7.0.3

Resolution

Stale snapshot entries associated to the <volume-id> are present in the VCDB CNS table and need to be removed.

Please, open a Support Request with the CNS team to get guidance on the resolution:

Support Product: VMware vCenter Server

Component Name: Cloud Native Storage