The RAV migration fails because the "ctkEnabled" parameter is enabled.
search cancel

The RAV migration fails because the "ctkEnabled" parameter is enabled.

book

Article ID: 423201

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

When attempting to migrate a Virtual Machine using HCX RAV, the migration fails during the switchover phase relocation.
Symptoms:

  • The error message in the HCX Manager app.log
    <timestamp> UTC [VmotionService_SvcThread-12035, Ent: HybridityAdmin, , TxId: <uuid>] ERROR c.v.v.h.m.common.MigrationUtil- [migId=<uuid>] Job (<uuid>) failed with exception System Error. Source side error is : Source side relocate failed for the virtual machine. A fatal internal error occurred. See the virtual machine's log for more details. msg.svmotion.fail.internal:A fatal internal error occurred. See the virtual machine's log for more details. msg.svmotion.file.createphase.fail:Failed to create one or more destination files. faultTime:<timestamp>
Target side error is : A general system error occurred: vMotion failed: unknown error msg.migrate.waitdata.platform:Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout.  vob.vmotion.recv.connection.closed:vMotion migration [168880629:8076383711162938946] the remote host closed the connection unexpectedly and migration has stopped. The closed connection probably results from a migration failure detected on the remote host. faultTime:<timestamp>
  • A log indicating that the files for the virtual machine to be migrated could not be read successfully is recorded in the mobilityagent.log.
    <timestamp> error mobilityagent[02195] [Originator@6876 sub=DiskLib opID=<uuid> user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-DSCPTR: DescriptorOpenInt: Cannot open disk descriptor "/vmfs/volumes/<datastore>/<vm-name>/<vm-name>-000001.vmdk" because it is an empty file. If this is in the context of s/xvmotion, it is most likely because the workflow initially creates empty destination disk descriptor files as placeholders.
<timestamp> error mobilityagent[02195] [Originator@6876 sub=DiskLib opID=<uuid> user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-DSCPTR: DescriptorOpen: Failed to open: The file specified is not a virtual disk
<timestamp> error mobilityagent[02195] [Originator@6876 sub=DiskLib opID=<uuid> user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LINK  : DiskLinkOpen: Failed to open '/vmfs/volumes/<datastore>/<vm-name>/<vm-name>-000001.vmdk': : The file specified is not a virtual disk
<timestamp> error mobilityagent[02195] [Originator@6876 sub=DiskLib opID=<uuid> user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-CHAIN : DiskChainOpen: "/vmfs/volumes/<datastore>/<vm-name>/<vm-name>-000001.vmdk": failed to open: The file specified is not a virtual disk.
  • In the source ESXi host vmware.log, the following entries are observed:
    <timestamp> In(05) vmx - Migrate: Caching migration error message list:
<timestamp> In(05) vmx - [msg.svmotion.fail.internal] A fatal internal error occurred. See the virtual machine's log for more details.
<timestamp> In(05) vmx - [msg.svmotion.file.createphase.fail] Failed to create one or more destination files.
<timestamp> In(05) vmx - Migrate: cleaning up migration state.
  • In the source ESXi host hostd.log, the following entries are observed:
    <timestamp> In(166) Hostd[2099626]: [Originator@6876 sub=Vcsvc.VMotionSrc.375375314443550950] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION
<timestamp> In(166) Hostd[2099626]: [Originator@6876 sub=Vcsvc.VMotionSrc.375375314443550950] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {
<timestamp> In(166) Hostd[2099457]: -->    faultMessage = (vmodl.LocalizableMessage) [
<timestamp> In(166) Hostd[2099457]: -->       (vmodl.LocalizableMessage) {
<timestamp> In(166) Hostd[2099457]: -->          key = "msg.svmotion.fail.internal", 
<timestamp> In(166) Hostd[2099457]: -->          message = "A fatal internal error occurred. See the virtual machine's log for more details.", 
<timestamp> In(166) Hostd[2099457]: -->       }, 
<timestamp> In(166) Hostd[2099457]: -->       (vmodl.LocalizableMessage) {
<timestamp> In(166) Hostd[2099457]: -->          key = "msg.svmotion.file.createphase.fail", 
<timestamp> In(166) Hostd[2099457]: -->          message = "Failed to create one or more destination files.", 
<timestamp> In(166) Hostd[2099457]: -->       }
<timestamp> In(166) Hostd[2099457]: -->    ], 
<timestamp> In(166) Hostd[2099457]: -->    reason = "A fatal internal error occurred. See the virtual machine's log for more details.", 
<timestamp> In(166) Hostd[2099457]: -->    msg = "A fatal internal error occurred. See the virtual machine's log for more details.
<timestamp> In(166) Hostd[2099457]: --> Failed to create one or more destination files.
<timestamp> In(166) Hostd[2099457]: --> "
<timestamp> In(166) Hostd[2099457]: --> }
  • Checking the vmx file of the virtual machine reveals that Changed Block Tracking (CBT) is enabled.
    The following example shows that while it is disabled for the entire VM, it is enabled for the disk.
    SCSI0:0.ctkEnabled = "True"
ctkEnabled = "FALSE"

Environment

VMware HCX

Cause

The migration failure is caused by the Change Block Tracking (CBT) parameters being inconsistently set or enabled on the virtual machine disks.
Specifically, if the ctkEnabled parameter is set to TRUE for the VM or its individual SCSI controllers/disks, it can interfere with the file creation phase on the target side during a RAV switchover in certain environment configurations.

Resolution

To resolve this issue, manually disable Change Block Tracking (CBT) on the affected Virtual Machine before re-attempting the migration.

1. Power off the virtual machine.
2. Right-click the virtual machine and click Edit Settings.
3. Click the Advanced Parameters tab.
4. Add or modify these entries:
 Set the ctkEnabled parameter value to false.
 Set the scsix:x.ctkEnabled parameter value to false.
Note: scsix:x in scsix:x.ctkEnabled indicates the SCSI device assigned to the hard disk that is added to the virtual machine.
Every hard disk added to the virtual machine is given a SCSI device that appears similar to scsi0:0, scsi0:1, or scsi 1:1.
5. Retry the HCX RAV migration.

Additional Information

For details on Changed Block Tracking (CBT), please refer to KB320557.
Changed Block Tracking (CBT) on virtual machines

Powered by Wolken Software