ERROR: “could not open/create change tracking files” when powering on VM

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• VM is unable to power on due to issues accessing CTK files, this can be a common occurrence after an abrupt storage outage.
• No CTK variables on the virtual machine configuration (.vmx).
• When the virtual machine is powered on, you see error similar to:
  
  
  invalid signature change in tracking file
  could not open/create change tracking file vmdk_file_name

Environment

VMware vCenter Server 6.x
VMware vCenter Server 7.x
VMware vCenter Server 8.x

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This issue occurs when the filename-ctk.vmdk file is not cleared.

Resolution

To resolve this issue, move the filename-ctk.vmdk file into a subdirectory or rename it utilizing the first method below, and if unsuccessful move to the second more detailed method. The third method has been added as an alternative way to handle VM's that have a lot of vmdk disks without disabling change tracking, or using the UI to modify the configuration. 

 

Method 1

 

1. SSH into the ESXi host where the affected VM resides utilizing root
2. Navigate to the VM's folder directory, this will be something along the lines of /vmfs/volumes/<DATASTORE>/<VM NAME>/
3. Create a temporary directory to place the files inside
   

   mkdir deleteme

4. Move the CTK files to the folder
   

   mv *ctk* ./TEMP

Method 2 

1. In the vCenter or ESXi host user interface, right-click the VM and select Edit Settings...
2. Make note of the SCSI controller type used by the hard disk.
3. Remove the vmdk_file_name hard disk specified in the error message from the virtual machine.

Warning: Select only Remove from virtual machine, Selecting Remove from virtual machine and delete files from disk will permanently delete the virtual disk.

4. Follow Using the ESXi command line to find the virtual machine file path and virtual disk information to connect to the ESXi host command line interface and navigate to the directory for the affected VM.
5. Move the filename-ctk.vmdk file into a subdirectory or rename it. Where the text in italics is replaced by the actual file name of the ctk file in the error:

    mkdir deleteme
    mv filename-ctk.vmdk deleteme

6. In the vSphere Client, re-add the hard disk back to the virtual machine using the SCSI controller type from step one.
7. On the ESXi host command line interface, edit the descriptor file (.vmdk) and remove or comment out any references that specify use of a ctk file.
8. Power on the virtual machine.

Method 3

1. Make sure HA is done attempting to restart the VM: 

   ps | grep -i <vm_name>

2. Make a temp directory in the vm's folder and move the change tracking files into the temp folder.

   mkdir temp
   mv *-ctk.vmdk temp

3. Use VI to comment out the changeTrackPath in each snapshot and base vmdk descriptor file. 

   # Change Tracking File
   #changeTrackPath="ExampleVM-ctk.vmdk"
   

4. Reload the virtual machines vmx file. Reloading a vmx file without removing the virtual machine from inventory

   vim-cmd vmsvc/getallvms | grep -i <vm-name>
   
   Vmid   Name           File                                   Guest OS                Version             Annotation
   10     ExampleVM      [datastore] ExampleVM/ExampleVM.vmx    windows8Server64Guest   vmx-08
   
   In the above example, the Vmid is 10.
   
   vim-cmd vmsvc/reload <Vmid>

5. Power on the virtual machine and perform consolidation to clean up the orphaned snapshots. Consolidating/Committing snapshots in VMware ESXi
   

Additional Information

For more information about Changed Block Tracking see:
Changed Block Tracking (CBT) on virtual machines

Further helpful KBs for VM power on issues:
Failed to power on virtual machine