HCX vMotion and Cold migrations are failing for some VMs with the error "Could not complete network copy for file"

Products

• VMware HCX

Issue/Introduction

• HCX vMotion/Cold migrations are failing only for some VMs. 
• The HCX /common/logs/admin/app.log shows the following error message. 

  <timestamp> UTC [VmotionService_SvcThread-######, Ent: HybridityAdmin, , TxId: ########-####-####-###########] ERROR c.v.h.s.v.j.MonitorSourceSideProgressWorkflow- [migId=########-####-####-###########] Remote relocate failure is observed from synced status from remote site. the message Target side relocate failed and error reason is Operation timed out.
  <timestamp> UTC [VmotionService_SvcThread-######, Ent: HybridityAdmin, , TxId: ########-####-####-###########] ERROR c.v.h.s.v.j.MonitorSourceSideProgressWorkflow- [migId=########-####-####-###########] Source side relocate 'task-######' failed for the virtual machine. Error is Could not complete network copy for file /vmfs/volumes/<Storage path>/<VMname>.vmdk. Total progress % is 'null'. com.vmware.vim.binding.vim.fault.NetworkCopyFault: Could not complete network copy for file /vmfs/volumes/<Storage path>/<VMname>.vmdk

• The /var/run/log/hostd.log of the ESXi server that hosts the virtual machine shows the following.   

  <timestamp> info hostd[2116677] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<Storage path>/<VMname>.vmx] Checking for all objects accessibility (VM's current state: VM_STATE_EMIGRATING, stable? false)
  <timestamp> info hostd[2116677] [Originator@6876 sub=Vcsvc.VMotionSrc.############] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION
  <timestamp> info hostd[2116677] [Originator@6876 sub=Vcsvc.VMotionSrc.############] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = (vmodl.LocalizableMessage) [
  -->       (vmodl.LocalizableMessage) {
  -->          key = "msg.svmotion.fail.internal",
  -->          arg = <unset>,
  -->          message = "A fatal internal error occurred. See the virtual machine's log for more details."
  -->       },
  -->       (vmodl.LocalizableMessage) {
  -->          key = "msg.svmotion.file.createphase.fail",
  -->          arg = <unset>,
  -->          message = "Failed to create one or more destination files."
  -->       }
  -->    ],
  -->    reason = "A fatal internal error occurred. See the virtual machine's log for more details."
  -->    msg = "A fatal internal error occurred. See the virtual machine's log for more details.
  --> Failed to create one or more destination files.
  --> "
  --> }

• /vmfs/volumes/<Data store path>/vmware.log shows that the file for the VM files copy failed. 

  <timestamp> | worker-185355937| I125: SVMotion: Enter Phase 4
  <timestamp> | vmx| W115: MigrateRPCHandleReply: Declaring RPC failure.
  <timestamp> | worker-185355937| W115: SVMotionRemoteDiskCreate: Remote disk create failed: 1
  <timestamp> | worker-185355937| W115: SVMotionPrepareForCopyThread: Failed to create destination files

• You can see the following "NFC ERROR" in the vpxa.log

  <timestamp> warning vpxa[2115382] [Originator@6876 sub=Libs opID=########-####-####-###########] [NFC ERROR] NfcFile_Open: Open failed:
  <timestamp>:05.698Z warning vpxa[2115382] [Originator@6876 sub=Libs opID=########-####-####-###########] [NFC ERROR] NfcFile_GetInfo: Failed to open file

Environment

VMware HCX

Cause

Both HCX vMotion and cold migration are failing because the underlying storage issue with the affected VM prevents the necessary disk copying.

Resolution

• Use bulk migration to avoid sVmotion. 
• Storage vMotion VM to a different storage and migrate using HCX.
  • If the disks are corrupted, this sVmotion operation will be unsuccessful, and the VM needs to be restored from a backup. 

Additional Information

"Could not complete network copy for file" also occurs when an issue or port is blocking NFC 902 TCP, or there is low network throughput. Ensure that you run the HCX Diagnostic to verify that port 902 is open, and also run the HCX perftest to ensure that it meets the minimum network requirements.

Refer to the Troubleshooting HCX Migrations Failures for additional information.