Failed to take snapshot: "number of snapshots on the source volume reaches the configured maximum (3)"

Products

• VMware vCenter Server

Issue/Introduction

• A snapshot-based backup of a CNS volume may fail if that volume has multiple snapshots associated with it. 

message: 'Failed to check and update snapshot content': failed to take snapshot  of the volume <volume-id> : "rpc error: code = FailedPrecondition desc = the number of snapshots on the source volume <volume-id>  reaches the configured maximum (3)"

Environment

• VMware vCenter Server 8.0
  
  
• Cloud Native Storage

Cause

• Old or stale snapshot entries associated to the <volume-id> are present in the VCDB CNS table and need to be removed.

Resolution

Follow the below steps to identify PVC/CNS volumes with multiple snapshots and delete them.

1. SSH into the vCenter appliance as root and log into the vCenter Database

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

2. Run the following query to get a list of CNS volumes with multiple snapshots 

select v.volume_name, v.volume_id, count(s.snapshot_id)
from cns.volume_info v
left join cns.vpx_storage_snapshot_info s on s.volume_id=v.volume_id
WHERE  v.is_cns_volume
GROUP by v.volume_name, v.volume_id
HAVING count(s.snapshot_id) >1;

3. Run the following query to get a list of volume_id and snapshots

select v.volume_id, s.snapshot_id
from cns.volume_info v
left join cns.vpx_storage_snapshot_info s on s.volume_id=v.volume_id
WHERE  v.is_cns_volume
and v.volume_name in (volume_name)
order by v.volume_id, s.create_time;

4. Login the the vCenter mob as administrator using the URL below (and using the vCenter id)

https://<vcenter address>/vslm/mob/?moid=VStorageObjectManager

5. Click on the VslmDeleteSnapshot_Task link, and this will pop up a small browser window. 

6. For each volume_id and snapshot_id  pair listed, populate the parameters and click "Invoke Method"

7. On the vCenter appliance, verify the tasks. Also, check that older snapshots are being removed by re-running the query in step 3.

Additional Information

• In rare cases, the vSphere inventory of managed virtual disks can become temporarily out of sync with datastore disk backing metadata. This problem can be due to a transient condition, such as an I/O error, or it can happen if a datastore is briefly inaccessible. This problem has been observed only under stress testing. Refer to article below for more details.

Reconciling Discrepancies in the Managed Virtual Disk Catalog

• For customers not using VKS, there is a freely available tool, CNS Manager, for customers to deploy which can scan and cleanup stale snapshots.

Orphan snapshots detection and clean-up

• There is also govc a command line tool, use to list CNS disks/volumes and snapshots. Refer to govmomi releases and installation steps.
  
  
• After removing PVC snapshots, attempts to snapshot the volume fail with the error "exceeded maximum number of permitted snapshots".  This error normally triggers when reaching the maximum number of snapshots per volume. Refer to article "exceeded maximum number of permitted snapshots" error persists after snapshots have been removed from Persistent Volume Claims