Resolving VM Consolidation Issues with Large Numbers of Snapshots (>32)

Products

VMware vSphere ESXi VMware vSAN

Issue/Introduction

Virtual machines (VMs) showing "Virtual Machine disk consolidation is needed" warnings fail to consolidate when they have accumulated a large number of snapshots (more than 32)
Standard consolidation attempts through vCenter fail
Cloning operations may also fail with "Too many levels of redo logs" errors.

Environment

VMware vSphere environments with VMs having more than 32 snapshots
Particularly affects VMs used with replication or backup solutions
Can occur with any VM regardless of operating system
vSAN OSA only, do not use this method for ESA clusters as ESA uses Native Snapshots and they're managed differently

Cause

While the system can technically create up to 254 snapshots for one virtual disk, VMware only supports up to 32 snapshots in a linear sequence (a single branch). To understand what we mean by a 'single branch', consider this example of a snapshot tree:

Base VM Virtual Disk
│
├── Snapshot 1
│ │
│ ├── Snapshot 2
│ │ │
│ │ ├── Snapshot 3
│ │ │ │
│ │ │ ├── Snapshot 4
│ │ │ │ │
│ │ │ │ └── Snapshot 5 <-- This is one branch with 5 snapshots
│ │ │ │
│ │ │ └── Alternative Snapshot Path 1
│ │ │
│ │ └── Alternative Snapshot Path 2
│ │
│ └── Alternative Snapshot Path 3

In this example, the path from Base VM through Snapshot 1, 2, 3, 4, and 5 represents a single branch. The vCenter and ESXi operating systems are designed to handle up to 32 snapshots in such a linear sequence for optimal performance and stability. While you may see other branches form (shown as Alternative Paths in the diagram), the 32-snapshot limit applies to any single path from the Base VM to the end of a branch.

The vCenter and ESXi operating systems therefore aren't designed to handle more than 32 snapshots in one branch.

Resolution

Power off the affected VM.
Disable Changed Block Tracking (CBT):
1. In vCenter, right-click VM > Edit settings... > Advanced Parameters
2. Filter the Attribute field for ctkEnabled
3. Set value to FALSE
Locate VM files and verify environment
1. Get VM ID and location
  
  vim-cmd vmsvc/getallvms |grep <vm_name>
  
  Example output:
  
  vim-cmd vmsvc/getallvms |grep testvm1
  17 testvm1 [datastore1] testvm1/testvm1.vmx windows8Server64Guest vmx-13
2. Get datastore path
  
  esxcli storage filesystem list |grep <datastore name>
  
  Example output:
  
  esxcli storage filesystem list |grep datastore1
  /vmfs/volumes/abcxxxxxx-xxxxxxxx datastore1 abcxxxxxx-xxxxxxxx true NFS 347967913943040 258073703899136
3. Change to VM directory
  
  cd /vmfs/volumes/abcxxxxxx-xxxxxxxx/testvm1
Create consolidated disk:
1. List files to find highest snapshot number
  
  ls -l
  
  Example output:
  -rw------- 1 root root 369 Jan 31 14:12 testvm1-000240.vmdk
  -rw------- 1 root root 64424509440 Jan 31 14:12 testvm1-000240-delta.vmdk
  -rw------- 1 root root 635 Jan 31 14:12 testvm1.vmdk
2. Create consolidated disk.
  
  vmkfstools -i "<path to VM><highest snapshot number>" "path to VM><vm name>_consolidated.vmdk" -d thin
  
  Example: Notice that we only use the vmdk descriptor file names in the command, not the -flat, -delta, or -sesparse file names:
  
  vmkfstools -i "/vmfs/volumes/abcxxxxxx-xxxxxxxx/testvm1/testvm1-000240.vmdk" "/vmfs/volumes/abcxxxxxx-xxxxxxxx/testvm1/testvm1_consolidated.vmdk" -d thin
Update VM configuration:
1. Backup VMX file
  
  cp <vmx file name> <vmx file name>.bak
  
  Example:
  
  cp testvm1.vmx testvm1.vmx.bak
2. Edit VMX file to repoint the VM to the new consolidated vmdk
  
  Example:
  
  vi testvm1.vmx
  
  # Find disk reference to the file you used in the consolidation command (press / then type):
  /testvm1-000240.vmdk
  
  # Where you find the the file change line from:
  scsi0:0.fileName = "testvm1-000240.vmdk"
  # To:
  scsi0:0.fileName = "testvm1_consolidated.vmdk"
  
  # Save and exit (press ESC, then type):
  :wq
Reload the vmx by running vim-cmd vmsvc/reload <Vmid> See KB Reloading a vmx file without removing the virtual machine from inventory for more details.

Validation Testing:

Power on VM
Create test snapshot:
1. Right-click VM > Snapshots > Take Snapshot
2. Enter name: "Test Snapshot"
3. Click OK
Delete test snapshot:
1. Right-click VM > Snapshots > Manage Snapshots
2. Select "Test Snapshot"
3. Click Delete
Verify consolidation:
1. Right-click VM > Snapshots > Consolidate
2. Verify task completes successfully
3. Check that "Virtual Machine disk consolidation is needed" warning is cleared

If Initial Consolidation Fails with below error then rerun the command with -v100 to get more details on why the process is failing.

Failed to open '/vmfs/volumes/Datastore/VM_Name/VM_Name-000254.vmdk': The parent of this virtual disk could not be opened (23). [root@esx004:/vmfs/volumes/Datastore/VM_Name]

If you see error "Too many open files" in the output stack then you will need to do an incremental consolidation.

Incremental Consolidation Method:
1. Consolidate first 30 snapshots
  
  vmkfstools -i "/vmfs/volumes/abcxxxxxx-xxxxxxxx/testvm1/testvm1-000030.vmdk" "/vmfs/volumes/abcxxxxxx-xxxxxxxx/testvm1/testvm1_consolidated30.vmdk" -d thin
2. Edit snapshot 31's descriptor to point to new base
  
  vi testvm1-000031.vmdk
  
  Original content example:
  
  # Disk DescriptorFile
  version=1
  encoding="UTF-8"
  CID=4b1240c1
  parentCID=4b1240c1
  createType="seSparse"
  parentFileNameHint="testvm1-000030.vmdk"
  
  Modified content:
  
  # Disk DescriptorFile
  version=1
  encoding="UTF-8"
  CID=4b1240c1
  parentCID=ffffffff
  createType="seSparse"
  parentFileNameHint="testvm1_consolidated30.vmdk"
3. Repeat process for subsequent chunks
  1. Consolidate snapshots 31-60:
    
    vmkfstools -i "/vmfs/volumes/abcxxxxxx-xxxxxxxx/testvm1/testvm1-000060.vmdk" "/vmfs/volumes/abcxxxxxx-xxxxxxxx/testvm1/testvm1_consolidated60.vmdk" -d thin
  2. Edit snapshot 61's descriptor
    
    vi testvm1-000061.vmdk
    ...
4. Continue this pattern for each 30-snapshot increment until reaching the final snapshot.
  Note: The amount of snapshot increment can be adjusted based on how many files can be opened before hitting the error "Too many open files"

Use VMware Converter:
1. Create new VM using VMware Converter
2. Migrate data to new VM
3. Decommission old VM