Troubleshooting virtual machine snapshot descriptor problems

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps for troubleshooting snapshot manager and descriptor ( .vmsd ) related problems. The snapshot descriptor file ( .vmsd) keeps the snapshot metadata and also maintains hierarchy between the snapshots.

If the snapshot descriptor gets corrupted information regarding snapshot becomes incorrect. This may lead to a situation where there are no snapshots in the snapshot manager window in VirtualCenter but the virtual machine keeps using delta disks which continue to grow and a vmfs volume may run out of free space.

Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0

Resolution

The process:

Check whether snapshots created by backup software
Verify the snapshot chains are healthy
Consolidate all the snapshots or Recreate the snapshot descriptor file from scratch

Check whether snapshots were created by backup software

Many backup solutions use vCenter Server as an intermediate and create a snapshot of the VM for backup.. When it is working properly, the snapshot created by this software always gets deleted.

To verify whether VM snapshots are created by backup software, look at the backup software state for the VM and compare the file date of backups to the file date of the snapshot files.

If the backup software is creating the snapshots, and failing to delete them, the datastore may fill up, or the number of supported snapshots may exceed limits. For further information, see Consolidating snapshot fails after third party backup succeeds

Ensure the snapshot chains are healthy

Assuming the missing snapshots were not created by the backup software, follow "The parent virtual disk has been modified since the child was created" error to check the health of the snapshot chains.

Consolidate all the snapshots

If the snapshot chains are healthy we need to consolidate the snapshots. There are three possible scenarios:

There is enough free disk space on the datastores containing the affected virtual disks to consolidate all the snapshots
There is not enough space on the datastores to consolidate the snapshots, but there is enough free disk space on other datastores to clone the disks
There is not enough space to either clone or commit the snapshots

Note: The space required to clone or commit the snapshots can vary depending on the size of the virtual snapshot disks, as well as the amount of changes that need to be committed, which can vary from almost none to almost 100% of the virtual snapshot disk size.

Scenario A: There is enough free disk space on the datastores containing the affected virtual disks to consolidate all the snapshots

Move the original .vmsd file to another directory, which we're naming "deleteme":

mkdir deleteme
mv *.vmsd deleteme

Create a new snapshot
Commit all snapshots using CLI or VI connected directly to ESX host ( not to Virtual Center ) where you replace the text in italics with the VM's ID you found above:

vim-cmd vmsvc/snapshot.removeall VM_ID

For more information, see Consolidate Snapshots
If there are problems with consolidation, see Consolidating/Committing snapshots in ESXi

Note: If the virtual machine is running during the consolidation process, it does need more disk space to process the consolidation. This is because a temporary consolidate helper disk is created to store all the changes during the process. The size of the disk depends on the I/O activity inside the virtual machine.

Scenario B: There is not enough space on the datastores to consolidate the snapshots, but there is enough free disk space on other datastores to clone the disks

Power off the virtual machine ( obligatory )
You should be able to clone the disks per Cloning and converting virtual machine disks with vmkfstools
Check what delta disk is currently in use

egrep -i "scsi[0-9]:[0-9].present|scsi.*filename" vm.vmx
Use vmkfstools to clone the snapshot disk

vmkfstools -i <snapshot.vmdk> <new-disk.vmdk>
Use the newly cloned disk
If the virtual machine has more then 1 disk repeat steps 3 & 4 for each disk
Power on the virtual machine

Note: Cloning is usually much faster then committing the snapshots, especially if there are a lot of snapshots in the chain.

Recreate the snapshot descriptor file

Scenario C: There is not enough space to either clone or commit the snapshots

Power off the virtual machine ( optional but recommended )
Restore the snapshot descriptor file (*.vmsd ) – see below
Reload the state of the virtual machine or un-register/re-register the virtual machine or restart mgmt-vmware
Start committing the snapshots from the oldest to the latest one - one after another. This way the snapshot will always be merged with pre-allocated base disk so no additional space is required.

You can Storage vMotion the VM or other VMs to another datastore
You can reduce memory reservations to reduce swap file space, or redirect VM swap files to another location per Swap File Location
You can add more space to the datastore the affected VM virtual disk is on by adding an extent to the VMFS volume.

To restore the snapshot descriptor file ( *.vmsd )

This procedure you haven't already moved or deleted the VM snapshot descriptor file (*.vmsd). If the file was already removed, please proceed starting with Step 5.

Connect to the ESXi host managing the VM by following
Find and navigate to the VM's directory by following Using the ESXi command line to find the virtual machine file path and virtual disk information
Create a sub-directory to move the old snapshot descriptor file (*.vmsd) into

mkdir deleteme

Move the descriptor file into the sub-directory

mv *.vmsd deleteme

Through the vCenter Server or ESXi host client user interface, create a new snapshot for the VM. This will create the new descriptor file.

Additional Information

Committing snapshots when there are no snapshot entries in the Snapshot Manager
Snapshot Best practices for vCenter Virtual Machines
Working with snapshots
Managing Snapshots
Snapshot Limitations
Failed to power on virtual machine