Performing vMotion fails at 78% with the error: Could not open change tracking file

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Performing vMotion fails at 78%
• You see this error:
  
  A general system error occurred. Source detected that destination failed to resume
  
  
• In the /vmfs/volumes/VMDatastore/VMFolder/vmware.log file, you see entries similar to:
  
  vmx| DISKLIB-LIB : Resuming change tracking.
vmx| DISKLIB-CTK : Could not open tracking file. File open returned IO error 4.
vmx| DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/4da7419d-########-####-##########ed/VM/VM-ctk.vmdk": Could not open/create change tracking file(2108).
vmx| DISK: Cannot open disk "/vmfs/volumes/4da7419d-########-####-##########ed/VM/VM.vmdk": Could not open/create change tracking file (2108).
vmx| Msg_Post: Error


• In the /var/log/hostd.log file, you see entries similar to:
  
  verbose 'vm:/vmfs/volumes/4da7419d-########-####-##########ed/VM/VM.vmdk'] VMotionLastStatusCb: Failed with error 7: Source detected that destination failed to resume.
verbose 'vm:/vmfs/volumes/4da7419d-########-####-##########ed/VM/VM.vmdk'] VMotionResolveCheck: Operation in progress
verbose 'vm:/vmfs/volumes/4da7419d-########-####-#########ed/VM/VM.vmdk'] Tools are auto-upgrade capable
verbose 'vm:/vmfs/volumes/4da7419d-########-####-##########ed/VM/VM.vmdk'] Tools guest daemon status changed to: 1
verbose 'vm:/vmfs/volumes/4da7419d-########-####-##########ed/VM/VM.vmdk'] VMotionStatusCb: Completed
verbose 'vm:/vmfs/volumes/4da7419d-########-####-##########ed/VM/VM.vmdk'] VMotionResolveCheck: Firing ResolveCb
info 'VMotionSrc (1307644538230054)'] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION
info 'VMotionSrc (1307644538230054)'] ResolveCb: Failed with fault: (vmodl.fault.SystemError) {
dynamicType = <unset>,
faultCause = (vmodl.MethodFault) null,
reason = "Source detected that destination failed to resume.",
msg = "",
}
verbose 'VMotionSrc (1307644538230054)'] Migration changed state from MIGRATING to DONE
verbose 'VMotionSrc (1307644538230054)'] Finish called

Environment

VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded
VMware ESX 4.1.x
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware ESXi 4.1.x Embedded

Cause

This issue occurs when you are using a backup application that uses Changed Block Tracking (CBT) and the ctkEnabled option for the virtual machine is enabled.

Resolution

To resolve this issue, you must remove the CTK variables from the virtual machine configuration (.vmx) file.

To remove the CTK variables from the .vmx file:

1. Log in to vCenter Server or the ESX/ESXi host using the vSphere Client.
2. Power off the virtual machine.
3. In the Inventory pane, right-click the virtual machine and click Edit Settings.
4. Go to Options > General.
5. Click Configuration Parameters.
6. Locate the ctkEnabled parameter and set it to false.
7. To guarantee that change block tracking cannot be re-enabled:
   
   1. Log into to the ESX service console.
   2. Power off the virtual machine.
   3. Open the .vmx file in a text editor.
   4. Add this line to the configuration file.
      
      ctkDisallowed="true"
      
      
   5. Save the file and close the editor.
      
      
8. Power on the virtual machine to apply the changes.

If the vMotion continues to fail, it may be necessary to remove the ctk.vmdk files from the virtual machine folder.

To remove the ctk.vmdk files:

1. Power off the virtual machine.
2. In the vSphere Client, select the virtual machine, click Summary, then go to Datastore > Browse Datastore.
3. Find the virtual machine in question, select the ctk.vmdk file, then right-click the files and select Delete selected items.
4. Edit the descriptor file (.vmdk) and remove or comment out any references that specify use of a ctk file
5. Power on the virtual machine and attempt the vMotion.

Note: If you are using VMware Data Recovery, ensure to reset the ctkEnabled parameter to true after the vMotion completes successfully.