Undetected Snapshots in Snapshot Manager and Snapshot Chain

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

For troubleshooting snapshots that are not detected by the snapshot manager and consolidating the snapshots when the consolidate option is grayed out in the Host UI and vCenter Web Client.  ​​​​VM is running on a snapshot but snapshots are not present in the snapshot manager 

• Space used by VMs do not add up to datastore total 
• Datastores filling up 
• Snapshots present in datastore are not in snapshot chain 

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0

Cause

Undetected snapshots can be caused by: 

• Insufficient space on the datastore during a snapshot creation, delete or consolidation 
• Failed API between the backup provider and VMware 
• Network or Storage event 
• The virtual machine shuts down abruptly during snapshot creation, delete, or consolidation

Resolution

Run a disk consolidation on the VM from the vCenter or Host Web Client. If the consolidation option is grayed out, try the following workarounds to get the VM to realize it needs consolidation. 

 

Option A - Recreate Snapshot Inventory and Tracking 

1. Create a temp subdirectory inside the affected VM directory. 
2. Move all files of affected VM ending in .vmsd, -ctk.vmdk, and .vmsn to the temp subdirectory. 
3. Reload the vmx file - https://knowledge.broadcom.com/external/article?legacyId=1026043(can be done powered on as no changes to the vmx file have been made) 
4. Create a new non-memory snapshot for the VM. This will recreate snapshot inventory and change tracking. 
5. Run a remove all from Snapshot Manager and consolidate from VM’s Actions menu. 

Note: The files moved in step 2 can be safely deleted ,once the VM is consolidated. 

 

Calculating consolidation progress - https://knowledge.broadcom.com/external/article/316414/how-to-calculate-snapshot-consolidation.html

 

 

Option B - Utilize vmkfstools to Clone a Disk 

1. Identify which disks are running on snapshots. 
2. Ensure there is sufficient space to create a copy of the disk(s). 
3. Power down the VM. 
4. Run the command  

"nohup vmkfstools -i /vmfs/volumes/Datastore1/Test/Test.vmdk /vmfs/volumes/Datastore1/Test_Clone.vmdk -d thin” 

Note: The nohup creates a text file that tracks the cloning process in the current running directory.  

5. Detach the disk with snapshots from the VM and attach the cloned disk. 
6. Power on the VM, and ensure the cloned disk is working as expected. 

Note: Once it is verified that the VM is running and stable with the cloned disk, the original disk and snapshots can be safely deleted . It is recommended to let the run for a few days to make sure there is no unexpected behavior on the cloned disk. 

 

Option C - Utilize V2V Converter to recreate the VM. (Vmware Standalone Converter has reached EOD and has been replaced with HCX .

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vcenter-converter/6-6/vmware-vcenter-converter-standalone-documenation/convert-a-physical-or-virtual-machine/select-a-source-machine-to-convert/select-a-hyper-v-server-virtual-machine.html

 

1. Download VMware Standalone Converter.VMware vCenter Converter download
2. Run a V2V conversion to recreate the VM. The new version of the VM will not have snapshots. 

Note: Once it is confirmed the converted machine is running and stable, the original VM with snapshots can be safely deleted. It is recommended to let the run for a few days to make sure there is no unexpected behavior on the cloned disk. 

Additional Information

Prevention 

As these snapshots will not show up in Snapshot Manager, we recommend checking periodically for these snapshots using vROPs, PowerCLI, RVTools, or the ESXi shell.  

*Run this command from the root directory of an ESXi host. This command will only check for snapshots in datastores that the ESXi host has access to.* 

find /vmfs/volumes/*/ -name "*000*.vmdk" | xargs ls -lh | awk '{ print $5,"\t"$6,$7,$8"\t",$9"\t",$10}'