[HCX] HCX Mobility Agent host randomly gets disconnected from vCenter Server after established Service Mesh

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

Symptoms:

• After established HCX Service Mesh from HCX Connector, Mobility Agent host in on-prem which is deployed by HCX Interconnect randomly reports "Host connection and power state" alarm in vCenter Server.

Alarm 'Host connection and power state' on <Mobility Agent IP> changed from Green to Red

• The vCenter Server vpxd log files contain entries similar to:

YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259F972700] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-4332c98c] [VpxdHostCnx] No heartbeats received from host; cnx: XXXX-XXXX-XXXX-XXXX-XXXX, h: host-4321, time since last heartbeat: 73361118066ms
YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259F972700] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-4332c98c] [VpxdHostCnx] Marking the connection alive to false: XXXX-XXXX-XXXX-XXXX-XXXX
YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259F972700] [Originator@6876 sub=InvtHostCnx opID=CheckforMissingHeartbeats-4332c98c] [VpxdInvtHost] Got lost connection callback for host-4321
YYYY-MM-DDT<TIME>+09:00 warning vpxd[7F259EDDB700] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-4321-b77f594] Connection not alive due to missing heartbeats; [vim.HostSystem:host-4321,172.XX.YY.ZZ], cnx: XXXX-XXXX-XXXX-XXXX-XXXX
YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259EDDB700] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-4321-b77f594] Succeeded restoring heartbeat; [vim.HostSystem:host-4321,<IP>]
YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259EDDB700] [Originator@6876 sub=MoHost opID=HostSync-host-4321-b77f594] [HostMo] host connection state changed to [CONNECTED] for host-4321

Cause

The Mobility Agent is a virtual host which is deployed by HCX Interconnect.
This issue will usually occur if vCenter Server is unable to communicate with Mobility Agent host via TCP443 and TCP/UDP 902 as same as typical ESXi hosts.

Resolution

Please check and reconfigure the firewall to allow access on the appropriate ports and make sure TCP443 and TCP/UDP 902 are not blocked between HCX Mobility Agent host and vCenter Server.

After validating the required ports, if the issue persists, you may re-deploy the HCX Interconnect.

Additional Information

VMware Ports and Protocols - VMware HCX