VM power-on fails with error: Change tracking invalid or disk in use

Products

VMware vSphere ESXi

Issue/Introduction

VM power on fails with generic error "Operation timed out". In some cases, it can take an extremely long time (over 24 hours) without even reaching the error message.

The vmware.log will report below errors with ctk files.

/vmfs/volumes/datastore/vm/vmware.log

vmx| I125: DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/12345678-111111111-abcd-efg1234567/ExampleVM_2/ExampleVM_2-000004-ctk.vmdk": Change tracking invalid or disk in use.
vmx| I125: DISKLIB-CTK : Change tracking invalid; reinitializing.
vmx| I125: DISKLIB-CTK : Auto blocksize for size 6442450944 is 4096.
'/vmfs/volumes/12345678-111111111-abcd-efg1234567/ExampleVM_2/ExampleVM_2_1.vmdk' (0xe): vmfs, 6442450944 sectors / 3 TB.
vmx| I125: OBJLIB-FILEBE : Error creating file '/vmfs/volumes/12345678-111111111-abcd-efg1234567/ExampleVM_2/ExampleVM_2-000004-ctk.vmdk': 3 (The file already exists).
vmx| I125: DISKLIB-CBT : Initializing ESX kernel change tracking for fid 15341702.
vmx| I125: DISKLIB-CBT : Successfully created cbt node ea1886-cbt.
vmx| I125: DISKLIB-CBT : Opening cbt node /vmfs/devices/cbt/ea1886-cbt
worker-2653492| I125: DISKLIB-LIB_BLOCKTRACK : Resuming change tracking.
worker-2653492| I125: DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/12345678-111111111-abcd-efg1234567/ExampleVM_2/ExampleVM_2-000004-ctk.vmdk": Change tracking invalid or disk in use.

Cause

This issue occurs when the filename-ctk.vmdk file is invalid.

Resolution

To resolve the issue we need to temporarily delete/move ctk files.

1. Before doing that we need to confirm that we don't have any locks using the below command:

# vmfsfilelockinfo -p ExampleVM_2-flat.vmdk

vmfsfilelockinfo Version 2.0
Looking for lock owners on "ExampleVM_2-flat.vmdk"
"ExampleVM_2-flat.vmdk" is locked in Read-Only mode by host having mac address ['##:##:##:##:##:##']
Trying to use information from VMFS Heartbeat
Host owning the lock on file is ###.###.###.###, lockMode : Read-Only
Total time taken : 14.671592491678894 seconds.

2. Check the VM process using the below command:

# esxcli vm process list |less

ExampleVM_2:
World ID: 1234567
Process ID: 0
VMX Cartel ID: 7654321
UUID: ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##
Display Name: ExampleVM_2
Config File: /vmfs/volumes/1aaaa1a1-12345678-a11a-aaa111a1abba/ExampleVM_2/ExampleVM_2.vmx

3. If there is a process, then kill it using the below command:

# esxcli vm process kill -t hard -w "world ID from the previous command"

4. Create a temp folder and move all the ctk files to that folder.

# mkdir temp
# mv *-ctk.vmdk

5. Backup the vmx file to the temp folder, the disable ctk mode by changing parameter from TRUE to FALSE - for all scsi entries.

Note: Disabling and enabling CBT will result in full backup

# cp vm-name.vmx vm-name.vmx.bck
scsix:x.ctkEnabled = "TRUE"----> scsix:x.ctkEnabled = "False"

6. Open all of the vmdks and disable ctk as well by putting # in the beginning of the line (changeTrackPath=...).

7. Remove the VM from inventory and re-add it back to reload the vmx file.

# vim-cmd vmsvc/reload VMID

8. Power on the VM. It should now power on without issues.

# vim-cmd vmsvc/power.on VMID

Note: The power on can take a long time. The progress bar might stay stuck at specific percentage for a long time, but the task is still running in the background.

Additional Information

Reference Articles: