HCX RAV and Bulk Migrations Fail with HttpHostConnectException on Port 8123

Products

• VMware HCX

Issue/Introduction

Following the deployment of a new VMware HCX Service Mesh, Data Plane diagnostics triggered from the UI indicate that port 8123 is blocked from the HCX Manager to the IX appliance. Replication Assisted vMotion (RAV) and Bulk migrations consistently fail to execute.

At the CLI level on the destination HCX manager, the app.log displays the following exception: com.vmware.vchs.hybridity.adapters.unified.hbr.server.fault.HbrConnectionException: Error connecting to HBR server <REDACTED_IP>:8123. Reason: HttpHostConnectException

NOTE: Went you attempt a migration at the UI level you will get the following error:

Error connecting to HBR server <REDACTED_IP>:8123. Reason: HttpHostConnectException

Executing the following command at the Mesh in question  netstat -plnt | grep -i hbrsrv and netstat -an | grep 8123 via SSH yields no output, confirming the service is not listening on the expected port. Furthermore, reviewing the configuration file /opt/vmware/etc/hbr/hbrsrv-nic.xml shows the ipForHMS bound to an incorrect IP rather than the expected Management IP.

root@<Obfuscated>-IXE-R1 :~ # cat /opt/vmware/etc/hbr/hbrsrv-nic.xml
<config>
<ipForHMS><Obfuscated_IP></ipForHMS>
<ipForNFC/>
<ipForFilter>127.0.0.1</ipForFilter>
</config>

Environment

VMware HCX 9.1

Cause

A race condition occurs while configuring hbrsrv, causing the service to bind and listen on the localhost address instead of the designated Management IP.

Resolution

This is a known Behavior in HCX 9.1 and will be resolved on  a later release.

Workaround 

 

1. SSH into the source HCX Manager.

2. Enter CCLI 

3. Execute list to view and identify the affected mesh (IX).

4. Execute go <affected ix number> to select the affected mesh.

5. After access the target appliance execute ssh to go into root

6. Restart the HBRSRV service by executing: systemctl restart hbrsrv

7. Repeat this procedure for the peer destination HCX environment.