Large VM stun times during snapshot operations on VMware vSAN

Products

• VMware vSphere ESXi
• VMware vSAN

Issue/Introduction

• • Virtual Machines (VMs) with large storage footprints (40TB to 150TB+) experience extended stun times, often exceeding 90 minutes, during snapshot creation or deletion.
  • The issue is specifically observed on vSAN.
  • The vmware.log for the affected VM shows that the majority of the stun duration is spent on Change Block Tracking (CBT) initialization or resets:
    • No (00) vmx ConfigDB: Setting ctkEnabled = "TRUE"
    • vcpu-0 DISKLIB-CBT : Initializing ESX kernel change tracking...
  • Backup performance is severely degraded during "Active Full" backup jobs that include a CBT reset.

Environment

• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x
• VMware vSAN OSA 8.x
• VMware vSAN ESA 8.x 

Cause

The initialization of Change Tracking (CTK) files for very large virtual disks requires significant metadata processing. When a CBT reset is triggered (either manually or by backup software), the system must perform a sequential re-scan of the allocation maps. For VMs with high disk fragmentation or massive capacity, this metadata overhead results in an extended stun window while the disks are closed and re-opened to initialize tracking.

Resolution

• The most effective solution to bypass this metadata bottleneck for large-scale VMs is to disable the automated CBT reset within the backup environment.

  1. Backup Job Configuration (Validated Workaround) To eliminate the stun condition, disable the CBT reset feature in your backup software.

  • Example (Veeam): Navigate to Edit Backup Job > Storage > vSphere > Advanced and uncheck "Reset CBT on each active full backup automatically."
  • Validation: In production testing, disabling this option allowed 150TB+ VMs to complete snapshots rapidly without noticeable stuns or unresponsiveness.

  2. Best Practices for Large VM Management

  • Virtual Disk Sizing: Aim to limit individual virtual disks to 5TB per VMDK. This keeps metadata processing windows within manageable limits.
  • Job Segmentation: Isolate VMs exceeding 35TB into dedicated backup jobs. This prevents long-running metadata operations from impacting the backup windows of smaller, critical production VMs.

  3. Long-Term Fix

  • Upgrade to VCF 9.1: Broadcom Engineering has introduced an enhancement in VCF 9.1 specifically designed to remove VM unresponsiveness during CBT-reset workflows. While the metadata operation may still take time, the enhancement prevents the VM from becoming unreachable/stunned during the process.

Additional Information

• When initiated by Backup tool, go to Edit Backup Job --> Storage --> vSphere --> Reset CBT on each Active Full backup automatically. (Validate this with your backup vendor, and uncheck this option before taking a full backup)