Throughput Drops to Zero During Backup Window (VM Snapshot Consolidation)
search cancel

Throughput Drops to Zero During Backup Window (VM Snapshot Consolidation)

book

Article ID: 417644

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms

  • Multiple virtual machines experience performance impact simultaneously.

  • Storage or application throughput (I/O) drops to 0 during the scheduled backup window.

  • Throughput degradation is observed during backup operations or snapshot consolidation.

  • Throughput returns appears normal once the backup or snapshot consolidation process completes.

  • No persistent performance degradation or data loss appears after the operation finishes.

        

 

Environment

VMware vSAN 8.x

Cause

  • This behavior is expected during the VMware snapshot consolidation phase.
    When a backup solution triggers snapshot creation and later merges (consolidates) delta files, VMware briefly stuns the virtual machine to ensure data consistency.

         Vmware.log :

  • 2025-10-27T13:41:08.150Z In(05) vmx - SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete.
    2025-10-27T13:41:08.590Z In(05) vcpu-0 - SnapshotVMXTakeSnapshotComplete: Done with snapshot '__G#_BACk##__': 489
    2025-10-28T13:33:06.256Z In(05) vcpu-0 - Checkpoint_Unstun: vm stopped for 385332 us
    2025-10-28T13:33:06.285Z In(05) vcpu-0 - CPT: vm was stunned for 414878 us
    2025-10-29T13:27:47.591Z In(05) vcpu-0 - Checkpoint_Unstun: vm stopped for 408370 us
    2025-10-29T13:27:47.620Z In(05) vcpu-0 - CPT: vm was stunned for 437796 us
    2025-10-29T13:48:37.386Z In(05) vcpu-0 - ConsolidateEnd: Snapshot consolidate complete: The operation completed successfully (0).

Resolution

This is expected behavior in VMware environments during snapshot consolidation. However, the impact can be mitigated by implementing the following best practices:

Schedule backups during off-peak hours to minimize production impact.

Reduce snapshot size by increasing backup frequency or retention rotation.

Verify storage performance ensure datastores and underlying storage can sustain high I/O throughput during consolidation.

Use Change Block Tracking (CBT)-enabled backup tools to reduce data delta sizes.

Monitor consolidation duration using vCenter performance metrics or VMware Aria Operations.

If the stun duration exceeds expected levels (seconds ,minutes), review snapshot size and storage latency for potential bottlenecks.

Additional Information

  • Consolidation stun duration is typically in milliseconds to a few seconds, depending on delta size and I/O speed.
  • Extended stuns may indicate:
  • High I/O workloads during backup
  • Oversized delta files
  • Latency in the storage subsystem
Powered by Wolken Software