Virtual machine experiences very long stun time during snapshot creation with CBT enabled
search cancel

Virtual machine experiences very long stun time during snapshot creation with CBT enabled

book

Article ID: 320346

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSAN

Issue/Introduction

  • Virtual machine experiences very long stun time during snapshot creation initiated from backup software.
  • The snapshot being created is non-quiesced and doesn't include virtual machine memory.
  • Changed Block Tracking is enabled on the virtual machine and relevant virtual disks devices before the Snapshot creation, but after the VM was last powered on.
  • Or Disks were hot added to the virtual machines when changed block tracking was disabled.
  • We see the following events in the Virtual Machine logs (vmware.log) for the affected virtual machine which shows that the majority of the stun time was spent on re-enabling the changed block tracking for the disk

    [YYYY-MM-DDTHH:MM:SS] No(00) vmx 68a4c935-01-e6-ece9 ConfigDB: Unsetting "ctkEnabled"
    [YYYY-MM-DDTHH:MM:SS]No(00) vmx b98bec7-01-6a-f276 ConfigDB: Setting ctkEnabled = "TRUE"

    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb Closing all the disks of the VM.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb Closing disk 'scsi0:3'
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CTK   : Auto blocksize for size 20971520000 is 16384.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 276525371.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Successfully created cbt node 107b713b-cbt.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/107b713b-cbt
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISK: Change tracking for disk 'scsi0:3' is now enabled.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Shutting down change tracking for untracked fid 276525371.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Successfully disconnected CBT node.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-VMFS  : "/vmfs/volumes/VMFS_VOLUME/VM_NAME/VM_NAME_6-flat.vmdk" : closed.

Environment

  • VMware vSphere ESXi 6.x
  • VMware vSphere ESXi 7.x
  • VMware vSphere ESXi 8.x
  • VMware vSAN OSA 8.x
  • VMware vSAN ESA 8.x 

Cause

Initializing changed block tracking on the virtual disks require the disks to be in unused stated (closed). 

When changed block tracking is enabled on a powered on virtual machine, it is not immediately enabled on the disks as the disks are in use. Rather it is deferred to next power cycle or snapshot creation/deletion time. In such cases depending on the size of the disks and data contained, initialization can take variable amount of time. 

When the snapshot is created for a virtual machine, disks are closed in order to create delta disks and attach them to the virtual machine. During this close/delta create/attach period, the virtual machine is stunned. In case of pending changed block tracking initialization this phase of snapshot creation will result in longer stun time.

Resolution

  • This is expected behavior
  • It is recommended to enable changed block tracking in a powered off state of virtual machine. Please read knowledgebase article Changed Block Tracking (CBT) on virtual machines
  • Enhancement to be added on version ESXi 9.1 to reduce the VM stun time in the snapshot when initializing CBT.

Additional Information

  • When initiated by Backup tool, go to Edit Backup Job --> Storage --> vSphere --> Reset CBT on each Active Full backup automatically. (Validate this with your backup vendor, and Uncheck this option before taking a full backup)
Powered by Wolken Software