Service Mesh creation failed with the error "Adding Mobility Agent Host failed. Cannot contact the specified host"
search cancel

Service Mesh creation failed with the error "Adding Mobility Agent Host failed. Cannot contact the specified host"

book

Article ID: 373065

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

When creating a Service Mesh in VMware HCX, the deployment fails during the interconnectConfigureMA workflow. This article outlines the steps to identify and resolve connectivity issues between the vCenter Server and the HCX Interconnect/Mobility Agent (IX/MA) appliance.

  • Creation of the service mesh fails. From HCX Manager UI -> Interconnect -> Service Mesh -> More -> Tasks, it's failing the error:
    "Adding Mobility Agent Host failed. Cannot contact the specified host".
    Service Mesh modification failed. Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (<IX-IP>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.
  • The following error is observed in /common/logs/admin/app.log:
    <timestamps> UTC [InterconnectService_SvcThread-240961, J:f807f56e, , TxId: ######-####-####-####-############] ERROR c.v.v.h.s.i.InitiateServiceMeshOperation- Failure detected while verifying completion of InterconnectServiceJobs::processServiceMesh. Reason: Interconnect Service Workflow processServiceMesh failed. Error: Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (<IX-IP>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding. java.lang.RuntimeException: Interconnect Service Workflow processServiceMesh failed. Error: Process Service Mesh failed. Interconnect Service Workflow interconnectConfigureMA failed. Error: Adding Mobility Agent Host failed. Cannot contact the specified host (<IX-IP>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.
  • vCenter var/log/vmware/vpxd.log logs shows below errors during MA host additional 
    <timestamps> error vpxd[94657] [Originator@6876 sub=Default] SSL Async Handshake Timeout : Read timeout after approximately 125000ms. Closing stream SSL(<io_obj p:0x00007f983082a870, h:74, <TCP '<VC-IP> : 42624'>, <TCP '<IX-IP> : 443'>>

Environment

VMware HCX
VMware vCenter Server

Cause

The deployment failure is primarily due to connectivity restrictions between the vCenter Server and the IX/MA appliance management interface. Common causes include:

  • Incorrect Network Profile: The Network Profile assigned to the Uplink/Management network is missing a valid Default Gateway.
  • Firewall Restrictions: Network security policies are blocking required management ports (TCP 443 and 902).
  • Asymmetric Routing: Network traffic is blocked by upstream devices due to routing inconsistencies.
  • Duplicate IP Address: Another device on the network is actively holding and using the IP address assigned to the IX appliance.

Resolution

To resolve this issue, perform the following troubleshooting steps:

  1. Verify Network Profile Configuration:
    • Log in to the HCX Manager UI.
    • Navigate to Infrastructure > Interconnect > Network Profiles.
    • Edit the Network Profile assigned to the Uplink/Management network.
    • Confirm that a valid Default Gateway is defined. If missing, configure the gateway, save the profile, and retry the Service Mesh operation.
  2. Validate Port Connectivity:
    • Log in to the vCenter Server Appliance (VCSA) console as root.
    • Test connectivity to the IX/MA appliance management IP using nc:
      nc -zv <IX_MA_IP> 443
nc -zv <IX_MA_IP> 902
    • Both commands must return a successful connection.
  3. Verify No Duplicate IP Addresses Exist on the Network:
    1. Start a continuous ping from the vCenter Server to the IX appliance IP address.
    2. Power off the IX appliance.
    3. Observe the ping results:
      • If the ping stops (times out): There is no duplicate IP address conflict on the network.
      • If the ping continues to reply: Another device on the network is actively using this IP address.
    4. If a duplicate IP is detected, investigate your network environment to identify the rogue device, resolve the IP conflict, and then power the IX appliance back on.
  4. Check Certificate Fetching:
    • Test if vCenter can fetch the IX appliance certificate:
      openssl s_client -connect <IX_MA_IP>:443
  5. Review Firewall and Routing:
    • If connectivity tests fail, review firewall or security group rules to ensure TCP ports 443 and 902 are permitted between the vCenter Server and the IX/MA management IP.
    • Check for asymmetric routing that may be blocking return traffic.
  6. Retry Deployment:
    • Once network configuration is verified, perform a Resync/Redeploy operation on the Service Mesh from the HCX Manager UI.

For more information on the required ports, refer to the VMware Ports and Protocols and Network Diagrams for VMware HCX.

Additional Information

Powered by Wolken Software