HCX vMotion and Cold Migrations Fail for All VMs with Errors: 'Could not complete network copy for file', and 'Cannot connect to host'

Products

• VMware HCX

Issue/Introduction

HCX vMotion and Cold Migrations are Failing for All VMs with the following error:

vMotion failed. System Error. Source side error is: Source side relocate failed for the virtual machine. Could not complete network copy for file /vmfs/volumes/vsan:[REDACTED]/HCX-test.vmdk Target side error is: Cannot connect to host.

Environment

VMware HCX

Cause

Duplicate HCX Mobility Agent entries in vCenter have been identified as the cause of the task failure within vCenter.

• Source HCX Manager /common/logs/admin/app.log shows the following error message.
  • YYYY-MM-DD HH:MM:SS.ms UTC [VmotionService_SvcThread-1678, Ent: HybridityAdmin, , TxId: ########-####-####-####-9ff2a4bee3ca] ERROR c.v.h.s.v.j.MonitorSourceSideProgressWorkflow- [migId=########-####-####-####-4ff1c2144487] Source side relocate 'task-####' failed for the virtual machine. Error is Could not complete network copy for file /vmfs/volumes/vsan:########-62a0191a8903e441/########-####-####-####-e4434b4e32b0/HCX-test.vmdk. Total progress % is 'null'.
com.vmware.vim.binding.vim.fault.NetworkCopyFault: Could not complete network copy for file /vmfs/volumes/vsan:########-62a0191a8903e441/########-a652-7dcb-507c-e4434b4e32b0/HCX-test.vmdk
    
    
• Destination HCX Manager /common/logs/admin/app.log shows the following error message.
  • YYYY-MM-DD HH:MM:SS.ms UTC [VmotionService_SvcThread-1649, Ent: HybridityAdmin, , TxId: TxId: ########-####-####-####-301de1554c35] ERROR c.v.h.s.v.j.MonitorTargetSideProgressWorkflow- [migId=########-####-####-####-4ff1c2144487] Target side relocate 'task-####' failed for the virtual machine. Error is Cannot connect to host.. Total progress % is null.
com.vmware.vim.binding.vim.fault.HostConnectFault: Cannot connect to host.
    
    
• Destination vCenter /var/log/vmware/vpxd/vpxd.log shows the following error message.
  • YYYY-MM-DD HH:MM:SS.msZ error vpxd[18851] [Originator@6876 sub=VmProv opID=4727-TxId: ########-####-####-####-301de1554c35-53-01] Failed to track task vim.Task:task-#### on host vim.HostSystem:host-227: Fault cause: vim.fault.HostConnectFault
-->
--> backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: #.#.#, build: build-########, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x013A4553]
--> backtrace[07] libvim-types.so[0x013B217E] Vim::Fault::HostConnectFault::ThrowInternal()

  • Run the below command on vCenter to check the Mobility Agent[MA] host entries. 

    • root@vcsa [ ~ ]# echo "select id,dns_name,ip_address from vpx_host where ip_address ='10.##.##.23';" | psql -d VCDB -U postgres
         id   |     dns_name      |   ip_address
--------+-------------------+-----------------
  host-240070   10.##.##.23  10.##.##.23
  host-227      10.##.##.23  10.##.##.23

    • Here, we can see that duplicate MA host entries caused the vpxd task to fail with a backtrace.

Resolution

Resolve the duplicate HCX Mobility Agent entries in vCenter by ensuring only one valid entry exists per IX appliance. 

• To identify the valid MA host ID from the list, you can refer to the screenshot below.
  • 
• Refer to the KB Manually removing an ESXi host from the vCenter Server database for detailed steps, and contact the vCenter team if additional assistance is needed.