HCX RAV, Cold, VMotion Migration fails with the error: "A specified parameter was not correct : Cannot connect to host"

Products

VMware HCX

Issue/Introduction

In the HCX UI > Migration > Mobility Group > Select the affected migration, the following error is seen:

vMotion failed. System Error. Source side error is : Source side relocate failed for the virtual machine. A specified parameter was not correct: Target side error is : Cannot connect to host.

In the HCX Manager, the following error message is observed in /common/logs/admin/app.log

<timestamp> UTC [ReplicationTransferService_SvcThread-5243, Ent: HybridityAdmin, , TxId: TxId: ########-####-####-####-#########] INFO c.v.h.s.r.j.ContinuousReplicationJob- Job (########-####-####-####-#########) ContinuousReplicationJob performing work in state:SYNC_STATUS_ON_REMOTE for transfer ########-####-####-####-#########<timestamp> UTC [VmotionService_SvcThread-1437, Ent: HybridityAdmin, , TxId: TxId: ########-####-####-####-##########] ERROR c.v.h.s.v.j.MonitorTargetSideProgressWorkflow- [migId=3########-####-####-####-##########] Target side relocate 'task-######' failed for the virtual machine. Error is Cannot connect to host.. Total progress % is null.com.vmware.vim.binding.vim.fault.HostConnectFault: Cannot connect to host.

In the HCX IX-I1 appliance logs, the following error message is observed in /var/log/vmware/mobilityagent.log:

<timestamp> info mobilityagent[02355] [Originator@6876 sub=Vmsvc opID=####-#######-####-####-####-#########-##-##-##-#### user=vpxuser:<USER-ID>] Vmsvc::CreateVm: Exception thrown vmodl.fault.SystemError
<timestamp> info mobilityagent[02355] [Originator@6876 sub=Vcsvc.LLPM opID=####-#######-####-####-####-#########-##-##-##-#### user=vpxuser:<USER-ID>] CreateVm: transient VM creation failed -- Fault cause: vmodl.fault.SystemError
-->
<timestamp> info mobilityagent[02355] [Originator@6876 sub=AdapterServer opID=####-#######-####-####-####-#########-##-##-##-#### user=vpxuser:<USER-ID>] AdapterServer caught exception: N5Vmomi5Fault15InvalidArgument9ExceptionE(Fault cause: vmodl.fault.InvalidArgument
--> )

In the HCX IX-R1 appliance logs, the following error message is observed in /var/log/vmware/mobilityagent.log:

<timestamp> info mobilityagent[02343] [Originator@6876 sub=Libs opID=####-TxId: ########-####-####-####-#########-##-##-##-#### user=vpxuser:<USERID] CnxOpenTCPSocket: Cannot connect to server <ESXi-IP>:902: Connection timed out
<timestamp> info mobilityagent[02343] [Originator@6876 sub=Libs opID=####-TxId: ########-####-####-####-#########-##-##-##-#### user=vpxuser:<USERID] CnxAuthdConnect: Returning false because CnxAuthdConnectTCP failed
<timestamp> info mobilityagent[02343] [Originator@6876 sub=Libs opID=####-TxId: ########-####-####-####-#########-##-##-##-#### user=vpxuser:<USERID] CnxConnectAuthd: Returning false because CnxAuthdConnect failed
<timestamp> info mobilityagent[02343] [Originator@6876 sub=Libs opID=####-TxId: ########-####-####-####-#########-##-##-##-#### user=vpxuser:<USERID] Cnx_Connect: Returning false because CnxConnectAuthd failed
<timestamp> info mobilityagent[02343] [Originator@6876 sub=Libs opID=####-TxId: ########-####-####-####-#########-##-##-##-#### user=vpxuser:<USERID] Cnx_Connect: Error message: Failed to connect to server <ESXi-IP>:902
<timestamp> warning mobilityagent[02343] [Originator@6876 sub=Libs opID=####-TxId: ########-####-####-####-#########-##-##-##-#### user=vpxuser:<USERID] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server <ESXi-IP>:902
<timestamp> error mobilityagent[02343] [Originator@6876 sub=NfcManager opID=####-TxId: ########-####-####-####-#########-##-##-##-#### user=vpxuser:<USERID] Unable to connect to NFC server: Failed to connect to server <ESXi-IP>:902
<timestamp> error mobilityagent[02343] [Originator@6876 sub=NfcManager opID=####-TxId: ########-####-####-####-#########-##-##-##-#### user=vpxuser:<USERID] Error encountered while opening clients for copy spec:

Cause

During the switchover process, the IX appliance connects to the ESXi host via the management network using port 902 (NFC) to transfer any remaining data. This error occurs when the IX appliance is unable to establish this connection.

Resolution

The IX Management interface must be able to access any of the ESXi VMkernel adapters set up for vSphere Replication or the ESXi Management network for NFC (TCP 902) connections. If the IP address of the VMkernel adapter is not reachable through the Default Gateway configured for the Management Network Profile, a static route must be established on the Compute Profile.
Check the MTU size between the IX Management network and the ESXi host vSphere Replication or Management Network vmkernel adapter.
Use the following commands to test the MTU size:(Ping with MTU size specified)
```
ping -i <Source-interface> <Dest-IP> -M do -s <MTU-Size>
```
Traceroute can be used if the traffic is routed:
```
traceroute <Dest-IP> -I --mtu <MTU-Size>
```

Tcpdump to analyze the packets:

tcpdump -i <source-interface> tcp port 902 and host <destination-IP> -ne

Curl command to analyze TCP connection establishment

curl --interface <source-interface> -v -k telnet://<dest-ip>:902 <- For NFC
curl --interface <source-interface> -v -k telnet://<dest-ip>:8000 <- For VMotion

Additional Information

For more information on the required ports, refer to the VMware Ports and Protocols and Network Diagrams for VMware HCX.
For more information about MTU in the HCX environment, refer to the Configuring MTU for VMWare HCX Component