Troubleshooting virtual machine snapshot descriptor problems

Troubleshooting virtual machine snapshot descriptor problems

book

Article ID: 341646

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps for troubleshooting snapshot manager and descriptor ( .vmsd ) related problems. The snapshot descriptor file ( .vmsd) keeps the snapshot metadata and also maintains hierarchy between the snapshots.

If the snapshot descriptor gets corrupted information regarding snapshot becomes incorrect. This may lead to a situation where there are no snapshots in the snapshot manager window in VirtualCenter but the virtual machine keeps using delta disks which continue to grow and a vmfs volume may run out of free space.


Environment

VMware vSphere ESXi 

Resolution

The process:

  1. Check whether snapshots created by backup software
  2. Verify the snapshot chains are healthy
  3. Consolidate all the snapshots or Recreate the snapshot descriptor file from scratch

Check whether snapshots were created by backup software

Many backup solutions use vCenter Server as an intermediate and create a snapshot of the VM for backup.. When it is working properly, the snapshot created by this software always gets deleted.
 
To verify whether VM snapshots are created by backup software, look at the backup software state for the VM and compare the file date of backups to the file date of the snapshot files.
 
If the backup software is creating the snapshots, and failing to delete them, the datastore may fill up, or the number of supported snapshots may exceed limits. For further information, see Consolidating snapshot fails after third party backup succeeds
 
Ensure the snapshot chains are healthy

Assuming the missing snapshots were not created by the backup software, follow "The parent virtual disk has been modified since the child was created" error to check the health of the snapshot chains.

Consolidate all the snapshots

If the snapshot chains are healthy we need to consolidate the snapshots. There are three possible scenarios:
  1. There is enough free disk space on the datastores containing the affected virtual disks to consolidate all the snapshots
  2. There is not enough space on the datastores to consolidate the snapshots, but there is enough free disk space on other datastores to clone the disks 
  3. There is not enough space to either clone or commit the snapshots

Note: The space required to clone or commit the snapshots can vary depending on the size of the virtual snapshot disks, as well as the amount of changes that need to be committed, which can vary from almost none to almost 100% of the virtual snapshot disk size.

 
Scenario A: There is enough free disk space on the datastores containing the affected virtual disks to consolidate all the snapshots
  1. Move the original .vmsd file to another directory, which we're naming "deleteme":
mkdir deleteme
mv *.vmsd deleteme
  1. Create a new snapshot
  2. Commit all snapshots using CLI or VI connected directly to ESX host ( not to Virtual Center ) where the text in italics is replaced with the VM's ID found above:

    vim-cmd vmsvc/snapshot.removeall VM_ID
Note: If the virtual machine is running during the consolidation process, it does need more disk space to process the consolidation. This is because a temporary consolidate helper disk is created to store all the changes during the process. The size of the disk depends on the I/O activity inside the virtual machine.
 
Scenario B: There is not enough space on the datastores to consolidate the snapshots, but there is enough free disk space on other datastores to clone the disks
  1. Power off the virtual machine ( obligatory )
    Tthe disks should clone per Cloning and converting virtual machine disks with vmkfstools
  2. Check what delta disk is currently in use

    egrep -i "scsi[0-9]:[0-9].present|scsi.*filename" vm.vmx
  3. Use vmkfstools to clone the snapshot disk

    vmkfstools -i <snapshot.vmdk> <new-disk.vmdk>
  4. Use the newly cloned disk
  5. If the virtual machine has more then 1 disk repeat steps 3 & 4 for each disk
  6. Power on the virtual machine
Note: Cloning is usually much faster then committing the snapshots, especially if there are a lot of snapshots in the chain.
 
Recreate the snapshot descriptor file
Scenario C: There is not enough space to either clone or commit the snapshots
  1. Power off the virtual machine ( optional but recommended )
  2. Restore the snapshot descriptor file (*.vmsd ) – see below
  3. Reload the state of the virtual machine or un-register/re-register the virtual machine or restart mgmt-vmware
  4. Start committing the snapshots from the oldest to the latest one - one after another. This way the snapshot will always be merged with pre-allocated base disk so no additional space is required.
  • Storage vMotion the VM or other VMs to another datastore
  • Reduce memory reservations to reduce swap file space, or redirect VM swap files to another location per Swap File Location
  • Add more space to the datastore the affected VM virtual disk is on by adding an extent to the VMFS volume.
 
To restore the snapshot descriptor file ( *.vmsd )
 
If the VM snapshot descriptor file (*.vmsd) already moved or deleted. If the file was already removed, please proceed starting with Step 5.
  1. Connect to the ESXi host managing the VM by following
  2. Find and navigate to the VM's directory by following Using the ESXi command line to find the virtual machine file path and virtual disk information
  3. Create a sub-directory to move the old snapshot descriptor file (*.vmsd) into
mkdir deleteme
  1. Move the descriptor file into the sub-directory
mv *.vmsd deleteme
  1. Through the vCenter Server or ESXi host client user interface, create a new snapshot for the VM. This will create the new descriptor file.

Additional Information

Powered by Wolken Software