Understanding iterative redolog consolidations during online snapshot deletion in VMware ESX

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

During a Delete Snapshot operation on a running virtual machine with heavy I/O:

• Virtual disk redologs (delta.vmdks) can be observed to be created, grow and be deleted iteratively.
• The virtual machine's Snapshot Manager shows a Consolidated Helper snapshot.
• The virtual machine's vmware*.log file shows:
  
  vmx| SnapshotVMXConsolidateOnlineCB: Another iteration is needed.

Environment

VMware ESX 4.1.x
VMware ESX Server 3.0.x
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Installable
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware ESX 4.0.x
VMware ESXi 3.5.x Installable
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 5.0
VMware ESX Server 3.5.x

Resolution

During a Delete Snapshot operation, the content of two virtual disks are merged, the redo log contents are consolidated into its parent.

It is necessary to store both the writes from the Guest OS and the writes from the consolidation operation for a running virtual machine. Further, the source of the consolidate must be read-only during the consolidation. This is referred to as an Online Consolidate.

ESX uses an iterative process to accommodate these requirements:

1. The size of the source redolog files are reviewed.
2. If the contents can be copied into the parent within a few seconds:
   
   
   1. The virtual machine is momentaraliy paused.
   2. Data is copied from the source redolog (eg, 00001) to its parent.
   3. The virtual machine resumes running from the parent.
      
      
3. If the contents cannot be copied into the parent within a few seconds:
   
   
   1. If this is not the first iteration, it is logged similar to:
      
      vmx| SnapshotVMXConsolidateOnlineCB: Another iteration is needed.
      
      
   2. A new snapshot is created and named "Consolidate Helper".
   3. An associated redolog (eg, 00002) is created and opened read-write.
   4. Data is copied from the source redolog (eg, 00001) to its parent.
   5. During the copy operation, the Guest OS continues to write new data to the read-write consolidate helper (eg, 00002).
   6. Data copy is completed, and the source redolog files (eg, 00001) are deleted.
   7. The size of the new redolog files (eg, 00002) are reviewed.
   8. The process repeats from step 1.
      
      

The general trend is that the amount of data in the additional redolog will decrease over time. As the amount of data being written by the Guest OS increases, the duration of a snapshot delete and redolog consolidation will also increase.

Note: There is no option to cancel the snapshot delete or redolog consolidation operations. To allow the snapshot delete to complete, reduce the amount of write disk I/O generated by the virtual machine's Guest OS and applications.

Additional Information

For more information, see Understanding virtual machine snapshots in VMware ESX (1015180).
Consolidating snapshots in ESX/ESXi 3.x and 4.x
Understanding VM snapshots in ESXi / ESX