VM becomes unresponsive for an extended period when change tracking file fails to open

Products

• VMware vSphere ESXi

Issue/Introduction

Changed Block Tracking (CBT) is enabled for the Virtual Machine (VM).

VM disk consolidation fails, and the "Consolidation Needed" status persists.

The VM may stun for an extended period (e.g., over 40 minutes), and the following error is observed in the vmware.log:

<YYYY-MM-DD>T##:04:38.###Z In(05) vcpu-0 - DISKLIB-CTK   : Could not open change tracking file "/vmfs/volumes/<DATASTORE>/<VM_NAME>/<VM_NAME>-000001-ctk.vmdk": Change tracking invalid or disk in use.
<YYYY-MM-DD>T##:04:38.###Z In(05) vcpu-0 - DISKLIB-CTK   : Change tracking invalid; reinitializing.
<YYYY-MM-DD>T##:04:38.###Z In(05) vcpu-0 - DISKLIB-CTK   : Auto blocksize for size #####52000 is 1024.
<YYYY-MM-DD>T##:04:38.###Z In(05) vcpu-0 - OBJLIB-FILEBE : Error creating file '/vmfs/volumes/<DATASTORE>/<VM_NAME>/<VM_NAME>-000001-ctk.vmdk': 3 (The file already exists).
<YYYY-MM-DD>T##:04:38.###Z In(05) vcpu-0 - DISKLIB-CBT   : Initializing ESX kernel change tracking for fid #####3287.
<YYYY-MM-DD>T##:04:38.###Z In(05) vcpu-0 - DISKLIB-CBT   : Successfuly created cbt node ####6877-cbt.
<YYYY-MM-DD>T##:04:38.###Z In(05) vcpu-0 - DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/####6877-cbt
<YYYY-MM-DD>T##:49:44.###Z In(05) vcpu-0 - DISKLIB-CBT   : Shutting down change tracking for untracked fid #####1458.
<YYYY-MM-DD>T##:49:44.###Z In(05) vcpu-0 - DISKLIB-CBT   : Successfully disconnected CBT node.
...
<YYYY-MM-DD>T##:49:45.###Z In(05) vcpu-0 - CPT: vm was stunned for 27######## us

Environment

VMware vSphere ESXi 8.0.x

Cause

When the VM attempts disk consolidation, it fails to open the change tracking file due to an invalid tracking state. This triggers a reinitialization, which may cause the VM to stun for an extended period. An enhancement request has been submitted to prevent VM stuns in this scenario in future releases.

Resolution

1. Power off the virtual machine.
2. Right-click the virtual machine and click Edit Settings.
3. Click the Advanced Parameters tab.
4. Set the ctkEnabled parameter to FALSE for the ctkEnabled setting for the VM and for each SCSI disk(s).
5. Create a temporary directory within the VM folder(s) and move the .vmsd file and any change tracking files into that directory:

   mkdir temp		
   mv *.vmsd temp
   mv *-ctk.vmdk temp

6. Use the vi editor to comment out the changeTrackPath line in each snapshot and base .vmdk descriptor file.
   

   # Change Tracking File
   #changeTrackPath="ExampleVM-ctk.vmdk"

7. Power on the virtual machine and perform a disk consolidation to clear any orphaned snapshots.

Note: Refer to the "Additional Information" section for further details.

Additional Information

Enabling or disabling Changed Block Tracking (CBT) on virtual machines

ERROR: “could not open/create change tracking files” when powering on VM