HCX RAV and Bulk Migrations Fail with HttpHostConnectException on Port 8123

Products

• VMware HCX
• VMware Cloud Foundation

Issue/Introduction

Following the deployment of a new VMware HCX Service Mesh, Data Plane diagnostics triggered from the UI indicate that port 8123 is blocked from the HCX Manager to the IX appliance. Replication Assisted vMotion (RAV) and Bulk migrations consistently fail to execute.

At the CLI level on the destination HCX manager, the app.log displays the following exception: com.vmware.vchs.hybridity.adapters.unified.hbr.server.fault.HbrConnectionException: Error connecting to HBR server <REDACTED_IP>:8123. Reason: HttpHostConnectException

NOTE: Went you attempt a migration at the UI level you will get the following error:

Error connecting to HBR server <REDACTED_IP>:8123. Reason: HttpHostConnectException

Environment

• VMware HCX 9.1
• VMware Cloud Foundation (VCF) 9.1

Cause

A race condition occurs while configuring hbrsrv, causing the service to bind and listen on the localhost address instead of the designated Management IP.

Resolution

Verification Steps Before applying the workaround, confirm the service status at the CLI of the HCX IX appliance:

1. SSH into the destination HCX Manager.
2. Run the following command to check if the HBR service is listening on the expected port: netstat -plnt | grep -i hbrsrv
3. Run the following command to check for active connections on port 8123: netstat -an | grep 8123
   Result: No output indicates the service is not listening on the expected port, confirming the issue.

4. Furthermore, reviewing the configuration file /opt/vmware/etc/hbr/hbrsrv-nic.xml shows the ipForHMSbound to an incorrect IP rather than the expected Management IP.

   root@<Obfuscated>-IXE-R1 :~ # cat /opt/vmware/etc/hbr/hbrsrv-nic.xml
   <config>
   <ipForHMS><Obfuscated_IP></ipForHMS>
   <ipForNFC/>
   <ipForFilter>127.0.0.1</ipForFilter>
   </config>

Workaround Procedure Apply the following steps to both the source and destination Service Meshes:

1. SSH into the HCX Manager.
2. Enter the CCLI (HCX Command Line Interface).
3. Execute list to view and identify the affected mesh (IX).
4. Execute go <mesh_id> to select the affected mesh.
5. Access the target appliance by executing ssh.
6. Restart the HBRSRV service by executing: systemctl restart hbrsrv
7. Repeat this procedure for the peer HCX environment.

Post-Workaround Verification Repeat the commands in the Verification Steps section. You should now see the service listening on the Management IP and port 8123. Attempt the migration again to confirm the error is resolved.

• Note: This is a known behavior in HCX 9.1 and will be resolved in a later release. If the issue persists, ensure that any necessary firewall ports are not blocking encryption traffic and that host trust is established.