HCX IX Mobility Agent host randomly gets disconnected from vCenter Server after established Service Mesh

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

• After established HCX Service Mesh from HCX Connector, Mobility Agent virtual host which is deployed by HCX Interconnect randomly reports "Host connection and power state" alarm in vCenter Server.
  

  Alarm 'Host connection and power state' on <Mobility Agent IP> changed from Green to Red

• vCenter UI shows the Mobility Agent hosts as disconnected or not responding:
  
  
  
• The vCenter Server vpxd log (/var/log/vmware/vpxd/vpxd.log) files contain entries similar to:
  

  YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259F972700] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-4332c98c] [VpxdHostCnx] No heartbeats received from host; cnx: ####-####-####-####-####, h: host-####, time since last heartbeat: 73361118066ms
  YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259F972700] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-4332c98c] [VpxdHostCnx] Marking the connection alive to false: ####-####-####-####-####
  YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259F972700] [Originator@6876 sub=InvtHostCnx opID=CheckforMissingHeartbeats-4332c98c] [VpxdInvtHost] Got lost connection callback for host-####
  YYYY-MM-DDT<TIME>+09:00 warning vpxd[7F259EDDB700] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-####-b77f594] Connection not alive due to missing heartbeats; [vim.HostSystem:host-####,172.##.##.##], cnx: ####-####-####-####-####
  YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259EDDB700] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-####-b77f594] Succeeded restoring heartbeat; [vim.HostSystem:host-####,<IP>]
  YYYY-MM-DDT<TIME>+09:00 info vpxd[7F259EDDB700] [Originator@6876 sub=MoHost opID=HostSync-host-####-b77f594] [HostMo] host connection state changed to [CONNECTED] for host-####

Environment

VMware HCX [Mobility Agent]
vCenter Server

Cause

The Mobility Agent is a virtual host which is deployed by HCX Interconnect.
This issue will usually occur if vCenter Server is unable to communicate with Mobility Agent host via TCP443 and TCP/UDP 902 , same as typical ESXi hosts.

Resolution

Please check and reconfigure the firewall to allow access on the appropriate ports and make sure TCP443 and TCP/UDP 902 are not blocked between HCX Mobility Agent host and vCenter Server.
After validating the required ports, if the issue persists, you may re-deploy the HCX Interconnect.

Additional Information

Please refer Ports and Protocols - VMware HCX  for more information.

ESXi host disconnects intermittently from vCenter Server

By default, ESXi hosts transmit heartbeats every 10 seconds, and vCenter expects to receive at least one within a 60-second window. If vCenter server does not receive the UDP heartbeat during this period, the host is marked as "not responding". This behavior may indicate network congestion, packet loss, or a misconfigured firewall between the ESXi host and the vCenter Server. 

If the host reconnects to the vCenter Server shortly after disconnecting, this is most likely a network interruption which interferes with heartbeat packets being sent from the host to vpxd.
If the host is not connecting to vCenter Server at all, the cause may be a permanent network interference or service instability.

Note: If the host consistently disconnects at 60-second intervals, it is a strong indication that UDP port 902 traffic from the ESXi host to the vCenter Server is being blocked most commonly by a firewall.