Undetected Snapshots in Snapshot Manager and Snapshot Chain

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

For troubleshooting snapshots that are not detected by the snapshot manager and consolidating the snapshots when the consolidate option is grayed out in the Host UI and vCenter Web Client. VM is running on a snapshot, but snapshots are not present in the snapshot manager.

Space used by VMs do not add up to datastore total
Datastores filling up
Snapshots present in datastore are not in snapshot chain

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Undetected snapshots can be caused by:

Insufficient space on the datastore during a snapshot creation, delete or consolidation
Failed API between the backup provider and VMware
Network or Storage event
The virtual machine shuts down abruptly during snapshot creation, delete, or consolidation

Resolution

Run a disk consolidation on the VM from the vCenter or Host Web Client. If the consolidation option is grayed out, try the following workarounds to get the VM to realize it needs consolidation.

Option A - Recreate Snapshot Inventory and Tracking

SSH to esxi host with root account.
Create a temp subdirectory inside the affected VM directory.
```
mkdir /vmfs/volumes/Example_Datastore/Example_VM/temp
```

Move all files of affected VM ending in .vmsd, -ctk.vmdk, and .vmsn to the temp subdirectory.

mv /vmfs/volumes/Example_Datastore/Example_VM/Example_VM.vmsd  /vmfs/volumes/Example_Datastore/Example_VM/temp

Reload the vmx file Reloading a vmx file without removing the virtual machine from inventory
Create a new non-memory snapshot for the VM. This will recreate snapshot inventory and change tracking.
Run a remove all from Snapshot Manager and consolidate from VM’s Actions menu.

Note: The files moved in step 2 can be safely deleted ,once the VM is consolidated.

Calculating consolidation progress - How to calculate Virtual Machine snapshot consolidation

Option B - Utilize vmkfstools to Clone a Disk

Identify which disks are running on snapshots.
Ensure there is sufficient space to create a copy of the disk(s).
Power down the VM.
Run the command.

Note : When you clone a disk, you have to initiate the clone from the last disk. Example: if the VM is pointing to Example_VM-000009.vmdk and you have to initiate the clone from Example_VM-000009.vmdk

nohup vmkfstools -i /vmfs/volumes/Datastore1/Example_VM/Example_VM-000009.vmdk /vmfs/volumes/Datastore1/Example_VM/Example_VM_Clone.vmdk -d thin

Note: The nohup creates a text file that tracks the cloning process in the current running directory.

Detach the disk with snapshots from the VM and attach the cloned disk.
Power on the VM, and ensure the cloned disk is working as expected.

Note: Once it is verified that the VM is running and stable with the cloned disk, the original disk and snapshots can be safely deleted . It is recommended to let the run for a few days to make sure there is no unexpected behavior on the cloned disk.

Option C - Utilize V2V Converter to recreate the VM. (VMware Standalone Converter has reached EOD and has been replaced with HCX .

Download VMware Standalone Converter.VMware vCenter Converter download
Run a V2V conversion to recreate the VM. The new version of the VM will not have snapshots. Using Converter Standalone with Virtual Machine Sources

Note: Once it is confirmed the converted machine is running and stable, the original VM with snapshots can be safely deleted. It is recommended to let the run for a few days to make sure there is no unexpected behavior on the cloned disk.

Additional Information

Prevention

As these snapshots will not show up in Snapshot Manager, we recommend checking periodically for these snapshots using vROPs, PowerCLI, RVTools, or the ESXi shell.

*Run this command from the root directory of an ESXi host. This command will only check for snapshots in datastores that the ESXi host has access to.*

find /vmfs/volumes/*/ -name "*000*.vmdk" | xargs ls -lh | awk '{ print $5,"\t"$6,$7,$8"\t",$9"\t",$10}'