VM power-on fails with error: Change tracking invalid or disk in use

Products

VMware vSphere ESXi

Issue/Introduction

Virtual Machine power-on task fails with "Operation Timed Out" error or remains stuck in progress status for extended duration (exceeding 24 hours) without completion.
The /vmfs/volumes/datastore/VM-Name/vmware.log on the ESXi host will report below errors with ctk files.

vmx| I125: DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name_2-000004-ctk.vmdk": Change tracking invalid or disk in use.
vmx| I125: DISKLIB-CTK : Change tracking invalid; reinitializing.
vmx| I125: DISKLIB-CTK : Auto blocksize for size 6442450944 is 4096.
'/vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name_2_1.vmdk' (0xe): vmfs, 6442450944 sectors / 3 TB.
vmx| I125: OBJLIB-FILEBE : Error creating file '/vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name_2-000004-ctk.vmdk': 3 (The file already exists).
vmx| I125: DISKLIB-CBT : Initializing ESX kernel change tracking for fid 15341702.
vmx| I125: DISKLIB-CBT : Successfully created cbt node ea1886-cbt.
vmx| I125: DISKLIB-CBT : Opening cbt node /vmfs/devices/cbt/XXXXX-cbt
worker-2653492| I125: DISKLIB-LIB_BLOCKTRACK : Resuming change tracking.
worker-2653492| I125: DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name_2-000004-ctk.vmdk": Change tracking invalid or disk in use.

Cause

This issue occurs when the filename-ctk.vmdk file is invalid.

Resolution

To resolve this issue temporarily, Delete/Move ctk files.

1. Verify that no file lock issues exist by executing the following command.

# vmfsfilelockinfo -p ExampleVM_2-flat.vmdk

vmfsfilelockinfo Version 2.0
Looking for lock owners on "VM-Name_2-flat.vmdk"
"VM-Name_2-flat.vmdk" is locked in Read-Only mode by host having mac address ['##:##:##:##:##:##']
Trying to use information from VMFS Heartbeat
Host owning the lock on file is ###.###.###.###, lockMode : Read-Only
Total time taken : 14.671592491678894 seconds.

2. Identify the active Virtual Machine process.

# esxcli vm process list |less

ExampleVM_2:
World ID: 1234567
Process ID: 0
VMX Cartel ID: 7654321
UUID: ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##
Display Name: ExampleVM_2
Config File: /vmfs/volumes/#######-#######-#####-#######/VM-Name/VM-Name.vmx

3. Terminate any existing process using the World ID obtained above.

# esxcli vm process kill -t hard -w "world ID from the previous command"

4. Relocate CTK files to a new temporary directory.

# mkdir temp
# mv *-ctk.vmdk temp

5. Create a backup of the .vmx file and disable CBT for all SCSI entries by switching the ctkEnabled value to "False".

Note: Disabling and enabling CBT will result in full backup failure

# cp vm-name.vmx vm-name.vmx.bak
scsix:x.ctkEnabled = "TRUE"----> scsix:x.ctkEnabled = "False"

6. Modify all .vmdk files to disable tracking by commenting out the changeTrackPath line.

7. Refresh the Virtual Machine configuration by reloading it within the inventory using the below commands.

# vim-cmd vmsvc/getall | grep -i VM-Name

# vim-cmd vmsvc/reload VMID

8. Power on the Virtual Machine from command line using the VMID.

# vim-cmd vmsvc/power.on VMID

Note: The Virtual Machine power on might take a long time and the progress indicator might appear frozen. However, the task will continue processing in the background.

Additional Information

Reference Articles: