Openshift MTV migrations from vSAN fail with "Invalid change tracker error code (7228)" and vmkfstools exit code 255

Products

• VMware vSphere ESXi

Issue/Introduction

When attempting to migrate virtual machine workloads from a vSphere vSAN datastore to Red Hat Openshift using MTV, the migration fails during the disk cloning stage.

The vsphere-xcopy-volume-populator.go or vmkfstools-wrapper.log reports the following errors:

• Failed to clone disk: Invalid change tracker error code (7228).
• clone task failed with exit code 255, stderr: Failed to clone disk: Input/output error (327689).
• Logs may also indicate that xcopy used reported: 0, forcing the migration to fall back to a host-based SSH copy method.

Environment

• Source: VMware vSphere ESXi (All Versions)
• Destination: Openshift
• Migration Tool: Openshift Migration Toolkit for Virtualization (MTV)

Cause

1. Disk Space Exhaustion: The vmkfstools-wrapper.log file on the ESXi host grows uncontrollably during the migration process, filling the /var volume to 100% capacity. This prevents the vmkfstools process from writing required metadata or temp files, leading to an I/O error (327689).
2. CBT Inconsistency: The virtual machine's Changed Block Tracking (CBT) state has become invalid or out of sync, resulting in the "Invalid change tracker error code (7228)" when the migration tool attempts to snapshot or clone the disk.

Resolution

To resolve this issue and allow the migration to complete successfully, perform the following maintenance on the source ESXi host and VM:

• Clear Host Log Volume:

  • Connect to the source ESXi host via SSH.
  • Truncate or delete the log file to free up space: echo > /var/log/vmkfstools-wrapper.log
  • Verify the /var volume is no longer at 100% using df -h.

• Reset Changed Block Tracking (CBT):

  • Power off the source VM.
  • Disable CBT in the VM configuration.
  • Browse the VM's datastore folder and delete all files ending in -ctk.vmdk.
  • Re-enable CBT in the VM configuration and power the VM back on to regenerate clean tracking files.

• Implement Automated Log Rotation:

  • To prevent the volume from filling up again during the migration of multiple VMs, create a temporary cron job on the ESXi host to clear the vmkfstools-wrapper.log during the migration. 

• Restart Management Agents:

  • Restart the ESXi management agents to ensure a clean state: /etc/init.d/hostd restart /etc/init.d/vpxa restart
• External Factors:
  • RedHat Openshift and/or storage array vendor support may need to be engaged if the limiting factors are also affected by the destination datastore.