Failed to take snapshot: number of snapshots on the source volume reaches the configured maximum (3)"

Products

• VMware vCenter Server

Issue/Introduction

A snapshot based  backup of a CNS volume  may fail if that volumes has multiple snapshots  associated with it. 

message: 'Failed to check and update snapshot content: failed to take snapshot  of the volume <volume-id> : "rpc error: code = FailedPrecondition desc = the number of snapshots on the source volume <volume-id>  reaches the configured maximum (3)"'

Environment

Support Product: VMware vCenter Server 8.0

Component Name: Cloud Native Storage

Cause

Old or stale snapshot entries associated to the <volume-id> are present in the VCDB CNS table and need to be removed.

Resolution

 

 Identify PVC/CNS volumes with multiple snapshots 

1. ssh into the vCenter appliance as root and log into the vCenter Database

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

2. Run the following query to get a list of  PVC/CNS volumes with multiple snapshots 

select v.volume_name, v.volume_id, count(s.snapshot_id)
from cns.volume_info v
left join cns.vpx_storage_snapshot_info s on s.volume_id=v.volume_id
WHERE  v.is_cns_volume
GROUP by v.volume_name, v.volume_id
HAVING count(s.snapshot_id) >1;

3. Run the following query to get a list of volume_id and snapshots

select v.volume_id, s.snapshot_id
from cns.volume_info v
left join cns.vpx_storage_snapshot_info s on s.volume_id=v.volume_id
WHERE  v.is_cns_volume
and v.volume_name in
(<list of volume names>)
order by v.volume_id, s.create_time;

4. Login the the vcenter mob as administrator using the url below (and using the vCenter id)

https://<vcenter address>/vslm/mob/?moid=VStorageObjectManager

5. Click on the VslmDeleteSnapshot_Task link and this will pop up a smaller browser window, see below. 

6. For each   volume_id and snapshot_id  pair listed, you need to populate the parameters and click "Invoke Method"

7.  You can check that the task has run in vCenter

8. On the vCenter appliance you can check these tasks. Also you can check that older snapshots are being removed by re-running the query in step 3.

 

Additional Information

Reconciling Discrepancies in the Managed Virtual Disk Catalog

For customers not using VKS, there is a freely available tool, CNS Manager, for customers to deploy which can scan and cleanup stale snapshots.

https://github.com/vmware-samples/cloud-native-storage-self-service-manager/blob/main/docs/book/features/orphan_snapshots.md

There is also govc a command line tool you can use to list CNS disks/volumes and snapshots
https://github.com/vmware/govmomi/releases

See this kb for an example of how to install and use https://knowledge.broadcom.com/external/article?articleNumber=320790

This is a useful write up by VMwaes Cormac Hogan
https://cormachogan.com/2021/04/06/task-delete-a-virtual-storage-object-reports-a-specified-parameter-was-not-correct/