HCX Bulk/RAV Migration not progressing beyond 0%

Products

VMware HCX

Issue/Introduction

HCX Bulk/RAV Migrations are not progressing beyond 0% with the state Base Sync Initiated.
There is no indication of any errors in the HCX UI, and the migration workflows remain actionable to be halted.

The following is observed continuously on the HCX /common/logs/admin/app.log

<timestamp> UTC [ReplicationTransferService_SvcThread-###, Ent: HybridityAdmin, , TxId: ########-####-####-####-############] INFO  c.v.h.s.r.j.ReplicationTransferMonitor- Source side replication status is: {"state":"syncing","progressInfo":{"progress":0,"bytes_transferred":0,"bytes_to_transfer":0,"checksum_total_bytes":53#######,"checksum_compared_bytes":0}} for vm: <VM Name> for transfer ########-####-####-####-############

The following is observed in the IX /var/log/vmware/hbrsrv.log

<timestamp> warning hbrsrv[01747] [Originator@6876 sub=HttpConnectionPool-000000 opID=########-####-####-####-############] Failed to get pooled connection; <cs p:################, TCP:ESX-IP:80>, (null), duration: 3071msec, N7Vmacore15SystemExceptionE(No route to host)
<timestamp> warning hbrsrv[04631] [Originator@6876 sub=IO.Connection opID=########-####-####-####-############] Failed to connect; <io_obj p:################, h:74, <TCP 'IX-IP : PORT'>, <TCP 'ESXI-IP : 443'>>, e: 113(No route to host), duration: 3071msec

The following is observed on the ESXi /var/log/vmkernel.log where the VM being migrated is currently residing:

######:2024-##-##T##:##:##.###Z cpu#:######)WARNING: Hbr: 5093: Failed to establish connection to [IX VR IP]:31031 (groupID=VRID-########-####-####-####-############): Timeout

This article provides information to help identify some of the issues that can cause HCX Bulk/RAV Migrations to not proceed beyond the initial stages and show 0% progress indefinitely.

Environment

VMware HCX

Cause

Required communication flows between HCX and the local infrastructure, vCenter/ESXi/NSX, are unavailable for the migration workflow to proceed. HCX will continuously attempt to establish communication without failing.

ESXi firewall blocking ports required for HCX Bulk/RAV migrations.
DFW blocking ports, which HCX Bulk/RAV Migrations require.
Local Management Traffic sent to external DFGW and being dropped due to MTU restrictions.
vSphere Replication is not enabled on the expected VMkernel adapter.
vSphere Replication NFC communication is only done through the Management Network for the IX appliance, even if there is a separate Network Profile configured for VR.
VMkernel adapter enabled for vSphere Replication is becoming unresponsive.

Resolution

HCX Bulk/RAV Migration requires TCP 80, 443, 31031, and 44046 communication between the ESXI Management VMkernel IP or VMkernel adapter configured for vSphere Replication on all the ESXi hosts in the Service Cluster at Source and Target to the IX appliance.
- From the ESXi host
```
nc -zv <IP Address of the IX vSphere replication> 31031
```
The IX Management interface must have a path to any of the VMkernel adapters configured for vSphere Replication NFC (TCP 902). If the VMkernel adapter IP cannot be reached through the Default Gateway configured for the Management Network Profile, then a static route must be configured on the Compute Profile.
If no VMkernel adapter is enabled for VR NFC, the ESXi host will default to the Management interface, but based on the local networking environment, it may be required to enable NFC on the ESXi host Management Interface.
Verify firewall settings on all ESXi hosts in the Service Cluster and DFW configuration on NSX to allow all the required communication ports.
Navigate to HCX Manager UI -> Interconnect > Service Mesh > Run Diagnostics and review the results for any errors. The diagnostics will test connectivity from the IX appliance to the required components (e.g., vCenter, ESXi hosts, etc.) and identify any issues related to network communication. If there are any errors related to closed ports, review the network and firewall configuration. For more information on the required ports, refer to the VMware Ports and Protocols and Network Diagrams for VMware HCX.

Additional Information

If the vMotion network is different from the Management network,

Edit Compute profile -> Click CONTINUE to select vMotion Network Profile -> click ADVANCED CONFIGURATIONS.
Add the static route for vMotion Network as the Network and Gateway for the vMotion network as the Next Hop accordingly.
Finish the compute profile and Resync the service mesh to apply the changes.
For HCX Health Check, access: HCX - Health Check and Best Practices

If the ESXi host is configured with a separate interface for vSphere Replication, that interface will be used to communicate with the IX Appliance. Kindly verify whether vSphere Replication is enabled on any of the interfaces on the ESXi host, and update the IX Network Configuration accordingly.