VM power-on fails with error: Change tracking invalid or disk in use

Products

VMware vSphere ESXi

Issue/Introduction

Virtual Machine power-on task fails with "Operation Timed Out" error or remains stuck in progress status for an extended duration (exceeding 24 hours) without completion.
This issue is frequently observed following failed backup job attempts (e.g., VEEAM) or unsuccessful snapshot consolidation operations, which result in stale VMFS file locks and invalid CTK (Change Block Tracking) files.
The /vmfs/volumes/datastore/VM-Name/vmware.log on the ESXi host will report the following errors with ctk files:

vmx| I125: DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name_2-000004-ctk.vmdk": Change tracking invalid or disk in use.
vmx| I125: DISKLIB-CTK : Change tracking invalid; reinitializing.
vmx| I125: DISKLIB-CTK : Auto blocksize for size 6442450944 is 4096.
'/vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name_2_1.vmdk' (0xe): vmfs, 6442450944 sectors / 3 TB.
vmx| I125: OBJLIB-FILEBE : Error creating file '/vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name_2-000004-ctk.vmdk': 3 (The file already exists).
vmx| I125: DISKLIB-CBT : Initializing ESX kernel change tracking for fid 15341702.
vmx| I125: DISKLIB-CBT : Successfully created cbt node ea1886-cbt.
vmx| I125: DISKLIB-CBT : Opening cbt node /vmfs/devices/cbt/XXXXX-cbt
worker-2653492| I125: DISKLIB-LIB_BLOCKTRACK : Resuming change tracking.
worker-2653492| I125: DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name_2-000004-ctk.vmdk": Change tracking invalid or disk in use.

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

This issue is typically caused by one or a combination of the following factors:

Stale VMFS File Locks: A hung ESXi kernel process or an interrupted snapshot task maintains an exclusive, stale lock on the VM delta disk (-sesparse.vmdk). This prevents the hypervisor from allocating the disk resources required for state changes or consolidation.
Hypervisor Initialization Failure: During the VM boot sequence, the hypervisor attempts to validate existing CTK mapping arrays. If these files are inaccessible, or locked, the hypervisor cannot complete the initialization, causing the power-on task to hang or fail.
CTK File Corruption: Change Block Tracking (-ctk.vmdk) files become corrupted or invalid, frequently occurring when a third-party backup operation is forcibly terminated or interrupted.

Resolution

Important Note: Before initiating troubleshooting steps, please verify the status of any associated backup jobs and ensure that any pending snapshot consolidation tasks are addressed.

Troubleshooting Steps:

Verify file locks: Ensure that no file lock issues exist by executing: vmfsfilelockinfo -p ExampleVM_2-flat.vmdk

vmfsfilelockinfo Version 2.0
Looking for lock owners on "VM-Name_2-flat.vmdk"
"VM-Name_2-flat.vmdk" is locked in Read-Only mode by host having mac address ['##:##:##:##:##:##']
Trying to use information from VMFS Heartbeat
Host owning the lock on file is ###.###.###.###, lockMode : Read-Only
Total time taken : 14.671592491678894 seconds.
Identify active VM process: esxcli vm process list | less

ExampleVM_2:
World ID: 1234567
Process ID: 0
VMX Cartel ID: 7654321
UUID: ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##
Display Name: ExampleVM_2
Config File: /vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name.vmx
Terminate existing process: Use the World ID obtained in step 2:

esxcli vm process kill -t hard -w <WorldID from the previous command>
Relocate CTK files: Move corrupted files to a temporary directory:

mkdir temp
mv *-ctk.vmdk temp
Create a backup of the .vmx file and disable CBT for all SCSI entries by switching the ctkEnabled value to "False".
Note: Disabling and enabling CBT will result in full backup.
cp vm-name.vmx vm-name.vmx.bak
scsix:x.ctkEnabled = "TRUE"----> scsix:x.ctkEnabled = "False"
Modify all .vmdk files to disable tracking by commenting out the changeTrackPath line.
Refresh the Virtual Machine configuration by reloading it within the inventory using the below commands:

vim-cmd vmsvc/getall | grep -i VM-Name
vim-cmd vmsvc/reload VMID
Power on the Virtual Machine from command line using the VMID:
vim-cmd vmsvc/power.on VMID

Note: The Virtual Machine power on might take a long time and the progress indicator might appear frozen. However, the task will continue processing in the background.

Additional Information

Reference Articles: