Consolidating/Committing snapshots in VMware ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Identifying the symptoms before considering the right steps to successfully consolidate the virtual machine snapshots

The most typical causes for VMs to show ‘Virtual Machine disks consolidation is needed’ Alert:

• Snapshots cannot be deleted/consolidated after completing backups.
• There is not enough space on the datastore to perform consolidation. VM disk/disks would be residing on the datastore which has less than 1 GB available space.
• Third-party backup application (Veeam, Unitrends, Dataprotect) has locked snapshot files and failed to remove the snapshot after completing backups or failed to initiate backups. 
• vCenter Server and the ESXi host connectivity issues. 
• When there are more than the VMware recommended number of snapshots, consolidation may fail. (VMware recommends only 32 as the maximum number of snapshots under best practices).
• When large snapshots are undergoing consolidation, VM may show unresponsive/frozen but the alert continues to show up. 

Check for the appropriate scenarios

1. Validate the error message observed while trying to consolidate snapshot.
   • Troubleshooting issues resulting from locked virtual disks.
   • "An error occurred while consolidating disks" when deleting one or more snapshots or consolidate disks.
   • Snapshot removal task stops at 99%.
   • Finding and listing virtual machine snapshots.
      
2. Was the snapshot created manually or by a backup appliance?
   • Failed to lock the file or One or more disks are busy deleting virtual machine snapshots.

Monitoring VMware Snapshots and Best practices

• How to monitor snapshot deletion using the vim-cmd command.
• Estimate the time required to consolidate virtual machine snapshots.
• Increasing the time limit on snapshot consolidation.
• Best practices for using VMware snapshots in the vSphere environment.

Environment

VMware vSphere ESXi 

Resolution

Overview

Typically, the snapshot file is in use by the virtual machine until the snapshot is consolidated/deleted. When a snapshot is deleted, any additional files in the hierarchy that are not identified by the Snapshot Manager are included in the commit process.
Sometimes, if the VMs show consolidation needed and no snapshots are present under the snapshot manager, creating a new snapshot and deleting it clears the entire hierarchy. This means that all snapshot files on the virtual machine are committed, then deleted.

A small amount of free space is required to create the new snapshots. If the virtual machine needs to remain running, more space must be allowed as the new snapshot grows (accepting new changes to the virtual disks) as the older snapshots commit.

• Checking if virtual machine consolidation is required with the Web Client
• Checking if virtual machine is running on a snapshot, via Command Line
• Commit snapshots using the vSphere Web Client and HTML5 Client 6.x/7.x/8.x
• Commit snapshots using the vSphere Client 6.0
• Commit snapshots from the ESXi command line

Checking if virtual machine consolidation is required with the Web Client

To check if virtual machine consolidation is required:

1. Select a vCenter Server host or a cluster and click the Virtual Machines tab.

2. Right-click any of the virtual machine and click Snapshot Manager/Manage snapshots, it would show "Needs Consolidation/Delete Snapshots".

   Note: A Configuration Issue warning is also displayed in the Summary tab of the virtual machine indicating that virtual machine disk consolidation is needed.

Alternatively, edit the settings of the virtual machine and see if any of the virtual disks are using a -00000X.vmdk file. If no disks are using -00000X.vmdk, this virtual machine is not using any of these files. Although unlikely, it is possible that another virtual machine is storing its snapshots in this directory. Check the other virtual machines. If none of them refer to these files, they can be safely erased.

Checking if virtual machine is running on a snapshot, via Command Line

Open the virtual machine configuration file (.vmx). Entries in the configuration file (.vmx) file related to virtual disks start with scsix:x. Perform this while connected to an ESXi host through DCUI or SSH. This command shows the information of the virtual disks of a virtual machine:

grep file /vmfs/volumes/datastoreName/VMname/VMname.vmx 

Example:

grep vmdk "/vmfs/volumes/lun1/TestVM/TestVM.vmx" 

scsi0:0.fileName = "TestVM-000001.vmdk"

Use the vim-cmd commands which is mentioned under the "Commit snapshots from the ESXi command line" section.

Commit snapshots using the vSphere Web Client and HTML5 Client 6.x/7.x/8.x

To commit all snapshots by using the vSphere Client:

1. Take a Snapshot. For more information, see the Take a Snapshot section in the vSphere Product Documentation .
2. Delete all Snapshots. For more information, see the Delete all Snapshots section in the vSphere Product Documentation.
3. Consolidate Snapshots. If the Delete All snapshots does not work, with vSphere 6.x/7.x try to consolidate snapshots. For more information, see the Consolidate Snapshots section in the vSphere Product Documentation.

Commit snapshots using the vSphere Client 6.0

To commit all snapshots by using the vSphere Client:

1. Take a Snapshot. For more information, see the Take a Snapshot section in the vSphere Product Documentation .
2. Delete all Snapshots. For more information, see the Delete all Snapshots section in the vSphere Product Documentation .

Note: If unsuccessful in attempting to consolidate using the Take a Snapshot then Delete all Snapshots technique, attempt the technique again, but this time select the Quiesce guest file system checkbox to enable quiesced snapshots when performing a new snapshot.

Commit snapshots from the ESXi command line

To commit all snapshot from the command line on ESXi:

1. Log into the ESXi host as root via the DCUI console or a SSH session.
2. Run this command to get a list of virtual machines and the Virtual Machine ID (vmid) for each virtual machine:
   
   vim-cmd vmsvc/getallvms
   
   The output appears similar to:
   
   Vmid Name File Guest OS Version Annotation
1 vm1 [datastore1] vm1/vm1.vmx windows7Server64Guest vmx-08
3 testvm [iscsi1] testvm/testvm.vmx winNetDatacenterGuest vmx-08
   
   Make a note of the Vmid for the specific virtual machine.
    
3. To verify if the snapshot exists, run this command and check the Snapshot Name, Snapshot Created On, and Snapshot State:
   
   vim-cmd vmsvc/snapshot.get [Vmid]
   
   The output appears similar to:
   
   Get Snapshot:
|-ROOT
--Snapshot Name : SnapshotTest
--Snapshot Desciption :
--Snapshot Created On : 11/27/2019 13:49:55
--Snapshot State : powered on
    
4. Run this command to create a new snapshot:
   
   vim-cmd vmsvc/snapshot.create [VmId] [snapshotName] [snapshotDescription] [includeMemory] [quiesced]
   
   For example, to create a snapshot on the virtual machine named testvm:
   
   vim-cmd vmsvc/snapshot.create 3 SnapshotTest AsnapshotDescription 0 0
   
   Note:
   • Enter in a name that will appear in the snapshot manager.
   • A file .vmsd may interfere with the creation or removal of the snapshots if a memory snapshot was left behind. For more information, see Overview of virtual machine snapshots in vSphere.
     
     
5. Run this command to remove all snapshots:
   
   vim-cmd vmsvc/snapshot.removeall [Vmid]

Additional Information

• The remove snapshot process can take a long time to complete if the snapshots are large.
  • To monitor the commit of snapshots, see Snapshot removal task stops at 99%.
  • The commit process does not display progress directly. As long as the date on the files continues to update, the process is working.
    Also, if the virtual machine is off, use the file * command to see if any files are in use. If any of the files return the error message can't read `filename' (Device or resource busy), they are locked by the VMkernel. The commit process is actively being performed on those files.
  • When the commit has completed successfully, there will be no -00000X.vmdk or -00000X-delta.vmdk files left unless they were not part of the snapshot tree. If unused files exist, they can be deleted.
  • To confirm that the commit succeeded, view the .vmx file, and verify that virtual disks are now pointing to a base disk (-flat.vmdk).
    
    
• If the snapshot removal/disk consolidation is completed and the .vmx file still contains a disk that is points to a snapshot file, the commit process failed.
  • If the attempt was made with the virtual machine powered on (running), plan a maintenance window to try removing the snapshot files again with the virtual machine powered off.
  • If the removal still fails when the VM is powered off, then use one of two methods to resolve the issue:
    • Option 1:  Clone the impacted virtual disk using vmkfstools -i.
      • The source file name is the current active -00000X.vmdk as identified in the .vmx file.
      • When the disk clone is complete, then point edit the VM settings to use the newly cloned disk.
      • The original base disk and snapshot tree can be deleted after the VM disk is repointed to the new cloned disk.
    • Option 2: Use a UI to clone the entire VM
      • Perform a clone operation on the virtual machine.
      • The cloned virtual machine retains the content of the associated snapshot disks at the time of cloning.
      • When the virtual machine has been cloned successfully, verify the operation of the new virtual machine and decommission the previous virtual machine.
        • For more information on cloning a virtual machine using the Web Client, see Clone an Existing Virtual Machine.
          
          
• This issue may be due to left over snapshots from the Virtual Machine Backup tool left unconsolidated, Please do follow the article Failed to lock the file or One or more disks are busy deleting vm snapshots.

Note: Cloning using the vSphere Client, vSphere Web Client or HTML5 Client requires vCenter Server and all applicable licenses and does not allow individual virtual disks to be selected when connected to the virtual machine. Relative to using the vmkfstools -i command to perform the same operation on a single virtual disk, consolidating snapshots using vSphere Clients clones all disks and may require more disk space.

See also

• Cloning and converting virtual machine disks with vmkfstools (343140).

Additional Information