Virtual machine experiences very long stun time during snapshot creation

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machine experiences very long stun time during snapshot creation initiated from backup software.
• The snapshot being created is non-quiesced and doesn't include virtual machine memory.
• Changed Block Tracking is enabled on the virtual machine and relevant virtual disks devices before the Snapshot creation, but after the VM was last powered on.
• Or Disks were hot added to the virtual machines when changed block tracking was disabled.
• We see the following events in the Virtual Machine logs (vmware.log) for the affected virtual machine which shows that the majority of the stun time was spent on re-enabling the changed block tracking for the disk

  [YYYY-MM-DDTHH:MM:SS] No(00) vmx 68a4c935-01-e6-ece9 ConfigDB: Unsetting "ctkEnabled"
  [YYYY-MM-DDTHH:MM:SS]No(00) vmx b98bec7-01-6a-f276 ConfigDB: Setting ctkEnabled = "TRUE"

  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb Closing all the disks of the VM.
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb Closing disk 'scsi0:3'
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CTK   : Auto blocksize for size 20971520000 is 16384.
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 276525371.
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Successfully created cbt node 107b713b-cbt.
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/107b713b-cbt
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISK: Change tracking for disk 'scsi0:3' is now enabled.
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Shutting down change tracking for untracked fid 276525371.
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Successfully disconnected CBT node.
  [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-VMFS  : "/vmfs/volumes/VMFS_VOLUME/VM_NAME/VM_NAME_6-flat.vmdk" : closed.

Environment

• VMware vSphere ESXi 6.x
• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Cause

Initializing changed block tracking on the virtual disks require the disks to be in unused stated (closed). 

When changed block tracking is enabled on a powered on virtual machine, it is not immediately enabled on the disks as the disks are in use. Rather it is deferred to next power cycle or snapshot creation/deletion time. In such cases depending on the size of the disks and data contained, initialization can take variable amount of time. 

When the snapshot is created for a virtual machine, disks are closed in order to create delta disks and attach them to the virtual machine. During this close/delta create/attach period, the virtual machine is stunned. In case of pending changed block tracking initialization this phase of snapshot creation will result in longer stun time.

Resolution

• This is expected behavior
• It is recommended to enable changed block tracking in a powered off state of virtual machine. Please read knowledgebase article Changed Block Tracking (CBT) on virtual machines

Additional Information

• When initiated by Backup tool, go to Edit Backup Job --> Storage --> vSphere --> Reset CBT on each Active Full backup automatically. (Validate this with your backup vendor, and Uncheck this option before taking a full backup)