HCX - Service Mesh creation failed with the error "Adding Mobility Agent Host failed. Cannot contact the specified host"

Products

• VMware HCX

Issue/Introduction

Symptoms:

• Creation of the Service Mesh fails. From HCX Manager UI -> Interconnect -> Service Mesh -> More -> Tasks, it's failing the error: "Cannot contact the specified host":

Service Mesh modification failed. Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (192.168.0.86). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

 

• The following error is observed in /common/logs/admin/app.log:

2024-07-25 10:44:33.686 UTC [InterconnectService_SvcThread-240961, J:f807f56e, , TxId: ######-####-####-####-############] ERROR c.v.v.h.s.i.InitiateServiceMeshOperation- Failure detected while verifying completion of InterconnectServiceJobs::processServiceMesh. Reason: Interconnect
Service Workflow processServiceMesh failed. Error: Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (192.168.0.86). The host may not be available on the network, a netw
ork configuration problem may exist, or the management services on this host may not be responding.
java.lang.RuntimeException: Interconnect Service Workflow processServiceMesh failed. Error: Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (192.168.0.86). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

Cause

During the process of adding a Mobility Agent (MA) host, HCX sends a request to vCenter, which then attempts to add the MA host.
This issue occurs when vCenter cannot communicate with the IX/MA management IP due to network port blocks, specifically on TCP 443 and TCP 902.

Resolution

Ensure that vCenter can communicate with the IX/MA appliance on ports 443 and 902.

To test port connectivity from vCenter to IX/MA:

1. Log in as root user through the VMware vCenter Server Appliance console.
2. Run this command on the vCenter Server Appliance:

   nc -zv <IX-IP> 443
nc -zv <IX-IP> 902

3. Both ports should be open.If there are any errors related to closed ports, review the network and firewall configuration. For more information on the required ports, refer to the VMware Ports and Protocols and Network Diagrams for VMware HCX.

Additional Information

Using Curl to test port connectivity in VMware vCenter Server Appliance