VM Power On operation gets stuck for a very long time without any progress due to CTK reset on huge VMDKs

Products

• VMware vSphere ESXi

Issue/Introduction

• Unable to Power On VMs with huge storage size (VMDKs in TBs) from vSphere Client or direct Host Client. The task hangs without any progress.
• The virtual machine log file shows the following entries:
  
  /vmfs/volumes/####-#####-####-###########/VM_Name/vmware.log

YYYY-MM-DDTHH:MM:SS.###Z| worker-######| I005: DISKLIB-CTK   : Could not open change tracking file "/vmfs/volumes/####-#####-####-###########/VM_Name/VM_Name-000001-ctk.vmdk": Change tracking invalid or disk in use.
YYYY-MM-DDTHH:MM:SS.###Z| worker-######| I005: DISKLIB-CTK   : Change tracking invalid; reinitializing.
YYYY-MM-DDTHH:MM:SS.###Z| worker-######| I005: DISKLIB-CTK   : Auto blocksize for size ############# is #####.
YYYY-MM-DDTHH:MM:SS.###Z| worker-######| I005: OBJLIB-FILEBE : Error creating file '/vmfs/volumes/####-#####-####-###########/VM_Name/VM_Name-000001-ctk.vmdk': 3 (The file already exists).
YYYY-MM-DDTHH:MM:SS.###Z| vmx| I005: UTIL: Change file descriptor limit from soft #####,hard ##### to soft #####,hard #####.

Environment

VMware vSphere ESXi 8.x.

Cause

• This issue is observed when the CTK files become invalid if the disk is not successfully closed and the changes are not updated in the CTK file. Therefore, CBT needs to be re-initialized to prevent any data loss.
• When re-enabling CBT, it needs to retrieve all changed blocks. It may take a long time to retrieve all changed blocks if the disk is huge with many fragments of allocation. The speed of the storage also affects the time.

Resolution

The Power On operation will complete once the CTK is successfully initialized. This might take a long time depending on the storage size of the virtual machine.

Workaround

If the task is taking a very long time and powering on the VM is critical, perform the following steps to power on the VM immediately:

1. Terminate the VM process using the esxcli command explained in the KB Powering off an unresponsive virtual machine on an ESXi host.
   
   esxcli vm process kill -t soft -w WorldID
   
   
2. Disable CBT on the virtual machine.
   1. Follow the steps from KB Changed Block Tracking (CBT) on virtual machines.
      1. Follow Step 3 & 4 under section "To check if a virtual disk has CBT enabled" to disable CBT at the VMX level and for each VMDK. 
3. Power on the virtual machine.
4. Consolidate any snapshots that exist on the virtual machine.
5. Enable CBT on the VM by following KB Enabling or disabling Changed Block Tracking (CBT) on virtual machines.