RAV migrations failing with error message: VMotion failed. Migration to host <###-###-###-###> failed with error Connection closed by remote host, possibly due to timeout

Products

• VMware HCX

Issue/Introduction

• The following error is observed in HCX UI > Migrations > Select Migration Group > Select the VM:
  vMotion failed. Migration to host <###-###-###-###> failed with error Connection closed by remote host, possibly due to timeout (195887167). msg.checkpoint.precopyfailure:Migration to host <###-###-###-###> failed with error Connection closed by remote host, possibly due to timeout (195887167). vob.vmotion.net.send.start.failed:vMotion migration [-1408####36:7973########4608776] failed to send init message to the remote host <###-###-###-###> vob.vmotion.net.rcv.failed.status.type:vMotion migration [-1408####36:7973########4608776] (0-7373######2144) failed to receive 68/68 bytes from the remote host <###-###-###-###>: Connection closed by remote host, possibly due to timeout faultTime:####-##-##T11:26:53.444866Z
  
  
• The following error is observed in the HCX Manager log file /common/logs/app.log:
<time_stamp> UTC [VmotionService_SvcThread-2835, Ent: HybridityAdmin, , TxId: ########-####-####-####-############] ERROR c.v.h.s.v.j.StartSourceSideRelocateVmWorkflow- [migId=########-####-####-####-############] Source side relocate failed for the virtual machine. error is Migration to host <Destination_host> failed with error Connection closed by remote host, possibly due to timeout (########).  msg.checkpoint.precopyfailure:Migration to host <Destination_host> failed with error Connection closed by remote host, possibly due to timeout (########).  vob.vmotion.net.send.start.failed:vMotion migration [<migration_id>] failed to send init message to the remote host <Destination_host> vob.vmotion.net.rcv.failed.status.type:vMotion migration [<migration_id>] (<migration_id>) failed to receive 68/68 bytes from the remote host <Destination_host>: Connection closed by remote host, possibly due to timeout faultTime:<time_stamp>
com.vmware.vim.binding.vim.fault.GenericVmConfigFault: Migration to host <Destination_host> failed with error Connection closed by remote host, possibly due to timeout (########).
  
  
• The following error is observed in the IX appliance log file /var/log/vmware/mobility-agent.log:
<time_stamp> info mobilityagent[02701] [Originator@6876 sub=VmotionWorker:38466] Vmotion failed: (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "vMotion failed: unknown error"
-->    msg = ""
--> }
  
  
• The following error is observed in the ESXi host log file /var/log/vmkernel.log:
  <time_stamp> In(182) vmkernel: cpu35:428238741)MigrateNet: 1791: <migration_id> S: Successfully bound connection to vmknic vmk2 - '<source_IP>'
<time_stamp> Wa(180) vmkwarning: cpu35:428238741)WARNING: VMotionSend: 727: <migration_id> S: (0-0x4321ab5382a0) failed to receive 68/68 bytes from the remote host <Destination_IP>: Connection closed by remote host, possibly due to timeout
<time_stamp> Wa(180) vmkwarning: cpu35:428238741)WARNING: VMotionSend: 1240: <migration_id> S: failed to send init message to the remote host <Destination_IP>
<time_stamp> In(182) vmkernel: cpu35:428238741)Migrate: 102: <migration_id> S: MigrateState: Failed
  
  

Cause

The error message indicates that the IX Appliance and the ESXi host are not able to establish a TCP/IP connection due to a timeout. 

 

Resolution

The following actions should be performed to confirm that the underlying networking is healthy

• Verify the data path configuration and make sure that ports 8000, 902, and 443 are allowed on the firewall/proxy. Reference: HCX Required Ports

• Run a diagnostic scan on the service mesh to make sure that no errors are reported. Reference: Run Data Plane Diagnostic