Storage vMotion or Cloning failure: Inconsistent CBT file (7228) - VMware vSphere

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

A Storage vMotion task for a virtual machine (VM) from one datastore to another fails at approximately 33%. This failure typically occurs when Changed Block Tracking (CBT) is enabled and the associated tracking metadata has become inconsistent or corrupted. This article provides the log signatures for this failure and the steps to reset the tracking mechanism.

Symptoms

• Failure occurring at 100% completion when using vmkfstools -i to clone disks from NFS to vSAN.
  
  
• "Could not complete network copy" error message during VM cloning.
  
  
• Issue observed on both running and powered-off virtual machines.
• The Storage vMotion task fails with the error: Relocate virtual machine "Error caused by file /vmfs/volumes/####/VM-FOLDER/####.vmdk".
  
  
• The vCenter vpxd.log contains the following error:
  
  YYYY-MM-DDTHH:MM:SS warning vpxd[####] [Originator@6876 sub=Vmomi opID=####] VMOMI activation LRO failed; << ####, , \>, vm-#####, vim.VirtualMachine.relocate>, N3Vim5Fault9FileFault9ExceptionE(Fault cause: vim.fault.FileFault
YYYY-MM-DDTHH:MM:SS error vpxd[####] [Originator@6876 sub=Default opID=####] [VpxLRO] -- ERROR task-#### -- vm-#### -- vim.VirtualMachine.relocate: :vim.fault.FileFault
  
  
• The ESXi hostd.log shows below traces:

YYYY-MM-DDTHH:MM:SS info hostd[####] [Originator@6876 sub=DiskLib opID=####] DISKLIB-LIB_CLONE : Failed to clone : Invalid change tracker error code (7228).
YYYY-MM-DDTHH:MM:SS warning hostd[####] [Originator@6876 sub=Libs opID=####] DiskLibCloneDone: Failed changetracker clone or resize: Invalid change tracker error code.
YYYY-MM-DDTHH:MM:SS warning hostd[####] [Originator@6876 sub=Libs opID=####] [NFC ERROR]Nfc_DiskLib_Clone: Failed to create VMFSEx2 disk /vmfs/volumes/####/VM-FOLDER/####.vmdk Invalid change tracker error code (7228).
YYYY-MM-DDTHH:MM:SS warning hostd[####] [Originator@6876 sub=Libs opID=####] [NFC ERROR]NfcFileDskClone: Failed to clone disk at destination /vmfs/volumes/####/VM-FOLDER/####.vmdk: Invalid change tracker error code (7228).
  

Environment

• ESXi 8.x
• ESX 9.x
• vCenter 8.x
• vCenter 9.x
• vSAN
• NFS

Cause

This issue occurs when a virtual machine has Changed Block Tracking (CBT) enabled and the change tracking file (-ctk.vmdk) is invalid, inconsistent, or corrupted.

Resolution

Disable Change tracking:

1. Power off the virtual machine
2. Right-click the virtual machine and select Edit Settings.
3. Navigate to the VM Options tab.
4. Expand the Advanced section and select Edit Configuration.
5. Locate the following parameters and set their values to false:
   
   
   • ctkEnabled → false
   • scsix:x.ctkEnabled → false (repeat for every hard disk assigned to the VM, (e.g., scsi0:0, scsi0:1)
6. Navigate to the virtual machine folder on the datastore and delete all files ending in -ctk.vmdk.
7. For each base VMDK and subsequent snapshot, open the descriptor file using the vi editor and comment out: changeTrackPath:
   
   
   • Example:

     Before 
# Change Tracking File
changeTrackPath="ExampleVM-ctk.vmdk"

     After
# Change Tracking File
# changeTrackPath="ExampleVM-ctk.vmdk"   
     
     
8. Power on the VM 
9. Perform a storage vMotion task on the VM.

Note: Disabling CBT causes the next backup to perform a full scan (stun/read) of the disk, as the incremental tracking data has been reset.

Additional Information

Enabling or disabling Changed Block Tracking (CBT) on virtual machines