Service Mesh creation failed with the error "Adding Mobility Agent Host failed. Cannot contact the specified host"

Products

• VMware HCX

Issue/Introduction

• Creation of the Service Mesh fails. From HCX Manager UI -> Interconnect -> Service Mesh -> More -> Tasks, it's failing the error: "Adding Mobility Agent Host failed. Cannot contact the specified host".

Service Mesh modification failed. Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (<IX-IP>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

• The following error is observed in /common/logs/admin/app.log:

<timestamps> UTC [InterconnectService_SvcThread-240961, J:f807f56e, , TxId: ######-####-####-####-############] ERROR c.v.v.h.s.i.InitiateServiceMeshOperation- Failure detected while verifying completion of InterconnectServiceJobs::processServiceMesh. Reason: Interconnect
Service Workflow processServiceMesh failed. Error: Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (<IX-IP>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.
java.lang.RuntimeException: Interconnect Service Workflow processServiceMesh failed. Error: Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (<IX-IP>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.


• vCenter logs shows below errors during MA host addtion [var/log/vmware/vpxd.log]
  <timestamps> error vpxd[94657] [Originator@6876 sub=Default] SSL Async Handshake Timeout : Read timeout after approximately 125000ms. Closing stream SSL(<io_obj p:0x00007f983082a870, h:74, <TCP '<VC-IP> : 42624'>, <TCP '<IX-IP> : 443'>>)

Environment

VMware HCX
VMware vCenter Server

Cause

When adding a Mobility Agent (MA) host, HCX sends a request to vCenter, which then initiates the process to add the MA host.
This issue arises when vCenter is unable to communicate with the IX/MA management IP—typically due to asymmetric routing, firewall restrictions, or blocked network ports, particularly TCP 443 and TCP 902.

Resolution

Ensure that vCenter can communicate with the IX/MA appliance on ports 443 and 902.

To test port connectivity from vCenter to IX/MA:

1. Log in as root user through the VMware vCenter Server Appliance console.
2. Run this command on the vCenter Server Appliance:

   nc -zv <IX-IP> 443
nc -zv <IX-IP> 902


3. Run below command to check if vCenter is able to fetch IX host certificate.
openssl s_client -connect <IX-IP>:443


4. Both ports [443/902] should be open and vCenter should be able to fetch IX certificate. If there are any errors related to closed ports, review the network and firewall configuration. For more information on the required ports, refer to the VMware Ports and Protocols and Network Diagrams for VMware HCX.

Additional Information

Using Curl to test port connectivity in VMware vCenter Server Appliance
HCX - Mobility Agent (MA) deployment fails when vCenter 'vpxd.certmgmt.mode' is set to 'custom' mode