Virtual machine experiences very long stun time during snapshot creation
search cancel

Virtual machine experiences very long stun time during snapshot creation

book

Article ID: 320346

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Virtual machine experiences very long stun time during snapshot creation initiated from backup software.
  • The snapshot being created is non-quiesced and doesn't include virtual machine memory.
  • Change Block Tracking is enabled on the virtual machine and relevant virtual disks devices before the Snapshot creation, but after the VM was last powered on
    • Or Disks were hot added to the virtual machines when change block tracking was disabled.
  • Virtual Machine logs (vmware.log) for the affected virtual machine suggests the majority of the stun time was spent on re-enabling the change block tracking for the disk
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb Closing all the disks of the VM.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb Closing disk 'scsi0:3'
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CTK   : Auto blocksize for size 20971520000 is 16384.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 276525371.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Successfuly created cbt node 107b713b-cbt.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/107b713b-cbt
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISK: Change tracking for disk 'scsi0:3' is now enabled.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Shutting down change tracking for untracked fid 276525371.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-CBT   : Successfully disconnected CBT node.
    [YYYY-MM-DDTHH:MM:SS] vcpu-0 5df670d8-d8-2afb DISKLIB-VMFS  : "/vmfs/volumes/VMFS_VOLUME/VM_NAME/VM_NAME_6-flat.vmdk" : closed.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Initializing change block tracking on the virtual disks require the disks to in unused stated (closed). 

When change block tracking is enabled on a powered on virtual machine, it is not immediately enabled on the disks as the disks are in use. Rather it is differed to next power cycle or snapshot creation/deletion time. In such cases depending on the size of the disks and data contained, initialization can take variable amount of time. 

When the the snapshot is created for a virtual machine, disks are closed in order to create delta disks and attach them to virtual machine. During this close/delta create/attach period, the virtual machine is stunned. In case of pending change block tracking initialization this phase of snapshot creation will result in longer stun time.

Resolution

Powered by Wolken Software