Online snapshot consolidation task appears to stall at specific percentages despite continuous backend disk I/O
search cancel

Online snapshot consolidation task appears to stall at specific percentages despite continuous backend disk I/O

book

Article ID: 441168

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

When performing a online snapshot deletion or consolidation task in vCenter Server, the task progress appears to hang at specific percentages (including 100%) for an extended period before finally moving forward or completing.

In the vmware.log file located in the target virtual machine's folder, it can be observed that the consolidation progress value repeats without changing, as shown in the example below:

YYYY-MM-DDThh:mm:ss.uuuZ| vcpu-0| I125: SnapshotVMXConsolidateOnlineCB: consolidation progress: 100%
YYYY-MM-DDThh:mm:ss.uuuZ| vcpu-0| I125: SnapshotVMXConsolidateOnlineCB: consolidation progress: 100%
YYYY-MM-DDThh:mm:ss.uuuZ| vcpu-0| I125: SnapshotVMXConsolidateOnlineCB: consolidation progress: 100%

Environment

VMware vSphere ESXi 8.0

Cause

This behavior is expected and is not an issue. The snapshot consolidation progress tracker calculates progress based on the ratio of consolidated physical data to the total size of the delta disk. Crucially, the delta disk size only reflects non-zero writes. If UNMAP commands or zeroed writes are issued by the Guest OS after the snapshot is created, they are represented merely as bits in the grain table. Because these unmapped areas and zeroed writes are not included in the delta disk size calculation, the progress tracker does not account for the time required to consolidate them.

This behavior can occur on any Guest OS type.

Example Scenario: If disk defragmentation runs within a Windows Guest OS while a snapshot is active, it generates a massive amount of UNMAPs or zeroed writes. The consolidation of these areas will continue in the background even after the physical data is consolidated, causing the progress percentage to stall.

 

Progress Tracking Example (Why Stalls Occur)

To understand why the progress bar stalls at specific percentages, consider a scenario where a VM has two delta disks being consolidated:

  • Delta Disk 1 Size (Non-zero writes): 4GB
  • Delta Disk 2 Size (Non-zero writes): 6GB
  • Total Tracker Baseline Size: 10GB

Assume that for both disks, 30% of the total tracked disk area consists of UNMAPs and zeroed writes. This means Delta Disk 1 actually contains ~5.71GB of total data to process (4GB physical + 1.71GB UNMAPs/zeros). Delta Disk 2 actually contains ~8.57GB of total data to process (6GB physical + 2.57GB UNMAPs/zeros).

How the progress tracker behaves during this consolidation:

  • Stalling at 40%: The process consolidates the 4GB of physical data from Delta Disk 1. The progress tracker sees 4GB out of 10GB completed and reports 40%. However, it must still consolidate the remaining 1.71GB of UNMAPs/zeros for this disk. The progress bar will appear stalled at 40% until this background work finishes.
  • Stalling at 100%: Once Disk 1 is fully complete, it moves to Disk 2. It consolidates the 6GB of physical data. The tracker sees 10GB out of 10GB completed and reports 100%. However, it must still consolidate the remaining 2.57GB of UNMAPs/zeros for Disk 2. The task will appear stalled at 100% until this final metadata is consolidated.

Resolution

Because this is functioning as designed, there are no configuration changes or workarounds available in ESXi/vCenter to alter how the consolidation progress is calculated. Therefore, do not cancel the task; simply wait for the status to change to Completed.

To verify that background write operations are still actively processing while the progress appears stalled, check the performance charts for the target virtual machine:

  1. In the vSphere Client, navigate to the target virtual machine.
  2. Select the Monitor tab, then click on [Performance] > [Advanced].
  3. Click the Chart Options button at the top of the chart view.
  4. Configure the following settings to view real-time metrics.
    1. Timespan: Real-time
    2. Chart Type: Line chart
    3. Counters: Select Read rate (KBps) and Write rate (KBps) under the Disk metric group.
  5. Click OK to apply the settings.
  6. Observe the chart. While Guest OS I/O can also increase these values, continuous activity in the Read rate (KBps) or Write rate (KBps) may indicate that the consolidation process is still actively running.

Note: To prevent prolonged consolidation times in the future, consider configuring the Guest OS to prevent heavy UNMAP operations (such as scheduled defragmentation or disk optimization) from running while a snapshot is present..

Powered by Wolken Software