Determining if there are Leftover Delta Files or Snapshots that VMware vSphere or Infrastructure Client Cannot Detect

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on determining whether there are leftover delta files or snapshots that VMware vSphere or Infrastructure Client cannot detect.

 

Symptoms

• Reporting tools, such as the df command or LUN properties in VMware vSphere or Infrastructure Client, report that a datastore has less space available than you expected.
• The VMware vSphere or Infrastructure Client does not detect all delta disks.
• Virtual machines are running on snapshots that are not detected by the VMware vSphere or Infrastructure Client.
• A vCenter Server alarm is triggered:
  
  Virtual machine disks consolidation is needed
   
• You see this notification in the vSphere Web Client:
   

  Configuration Issues

  Virtual machine disks consolidation is needed

Environment

VMware ESXi 4.0.x Installable
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1
VMware ESX Server 3.5.x
VMware ESX Server 3.0.x
VMware ESX 4.0.x
VMware ESX 4.1.x
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 6.5
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 6.0
VMware ESXi 3.5.x Installable
VMware vSphere ESXi 6.7
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.0

Cause

This issue occurs because the delta files are not being cleaned up due to an API integration failure with virtual machine backup software. In this case, contact the backup software vendor.

Resolution

Virtual machines may be running in a snapshot state with delta disks that are not detected by the VMware vSphere or Virtual Center Client.

 

To verify the virtual machines that have unlisted snapshots attached:

1. Open the vCenter Server or Virtual Infrastructure Client and log in.
2. Select a virtual machine in your host inventory and click Edit Settings.
3. Review the virtual machine's hardware configuration, highlighting each attached virtual disk. Any disk filename containing a suffix similar to -000001.vmdk is indicative of the virtual machine running on snapshot delta disks.

To review virtual machine directory content using the vSphere Management Assistant (vMA) or vSphere CLI (vCLI):

 

 

Note: For more information about steps using the vMA or vCLI, see either vSphere Management Assistant or vSphere Command-Line Interface.

The above file listing suggests that delta disk files reside in the virtual machine's directory. Confirm that they are selected for use by reviewing the virtual machine's configuration in the VMware vSphere or Infrastructure Client. If they are not referenced, and only the base disk is selected, the delta disk files can be removed.

For more information, see Consolidating/Committing snapshots in ESXi (1002310)

 

To check virtual machine configurations for references to delta disk files via the ESX host terminal:

1. Log in to the ESX host terminal directly, or via SSH. For support on ESXi hosts, see Contact Broadcom support.
2. Run this command to search for all of the virtual machine configuration files seen by the ESX and check if there is a file which looks like a snapshot disk:
   
   find /vmfs/volumes/ -iname "*.vmx" -exec grep -HEie "-[0-9]{6}.vmdk" {} \;
   
   You see output similar to:
   
   /vmfs/volumes/49d9f2d8-########-####-##########e6/examplevm/examplevm.vmx:scsi0:0.fileName = "examplevm-000002.vmdk"
   /vmfs/volumes/49d9f2d8-########-####-##########e6/examplevm/examplevm.vmx:scsi0:1.filename = "examplevm_1-000002.vmdk"
   /vmfs/volumes/49d9f2d8-########-####-##########e6/examplevm/examplevm.vmx:scsi0:2.filename = "examplevm_2-000002.vmdk"
   /vmfs/volumes/49d9f2d8-########-####-##########e6/anothervm/anothervm.vmx:scsi0:0.fileName = "anothervm-000001.vmdk
   
   Note: This command must be run on each VMware ESX host. Virtual machine configuration files in-use by other hosts are not readable by the server running this command.

For an overview of what delta disk files are currently in-use by running virtual machines on an individual VMware ESXi/ESX 3.5, 4.0 or 4.1 host:

1. Log in to the ESX host terminal directly or via SSH. For support on ESXi hosts, see Contact Broadcom support.
2. List delta disks currently open by the respective VMware ESX host using:
   
   ls -l /vmfs/devices/deltadisks
   
   You see output similar to:
   
   total 134217728
   -rw------- 1 root root 107374182400 Sep 7 16:44 34baac8f-VMware Capacity IQ_1-000001-delta.vmdk
   -rw------- 1 root root 21474836480 Sep 7 16:44 7f9d62c9-examplevm-000003-delta.vmdk
   -rw------- 1 root root 8589934592 Sep 7 16:44 b750727-VMware Capacity IQ-000001-delta.vmdk
   -r--r--r-- 1 root root 0 Sep 7 16:44 control

For virtual machines that are not reported to have snapshots in the VMware vSphere or Infrastructure Client, commit or remove the snapshot delta disks. For more information, see Consolidating/Committing snapshots in ESXi (1002310)

If the delta disk files are not used by any virtual machine, you can delete them with caution. For more information, see Consolidating/Committing snapshots in ESXi (1002310) ​​​​​​​

 

Notes:

• The same vmkfstools -U command can be run within the ESX host terminal to remove the delta disks, if you are not using the VMware vMA or vCLI.
• Attempt the deletions while the respective virtual machine is powered on. If the files are in fact in-use and required by the virtual machine, the operation fails and prevent deletion of the disk files. Attempt to consolidate the respective delta disk file if this occurs.

To list all delta disk files stored across the datastores:

1. Log in to the ESX host terminal directly or via SSH. For ESXi hosts, see Contact Broadcom support.
2. List delta disks stored across the datastores using:
   
   find /vmfs/volumes/ -iname "*delta.vmdk"
   
   You see output similar to:
   
   /vmfs/volumes/49d9f2d8-########-####-##########e6/ examplevm/ examplevm-000001-delta.vmdk
   /vmfs/volumes/49d9f2d8-########-####-##########e6/ examplevm/ examplevm-000002-delta.vmdk
   /vmfs/volumes/49d9f2d8-########-####-##########e6/ examplevm/ examplevm-000003-delta.vmdk
   /vmfs/volumes/49d9f2d8-########-####-##########e6/ stalevm/ stalevm-000001-delta.vmdk
   /vmfs/volumes/49d9f2d8-########-####-##########e6/stale vm/stale vm-000002-delta.vmdk
   /vmfs/volumes/49d9f2d8-########-####-##########e6/anothervm/anothervm-000001-delta.vmdk

Note: In this example, examplevm is a legitimate virtual machine with three snapshot delta disks. stalevm is a decommissioned virtual machine that was removed from inventory, but not deleted from the datastore, unnecessarily taking up space.