Interconnect Service Mesh Tunnel State Change

Products

• VMware HCX

Issue/Introduction

 

• Service Mesh IX appliances are receiving an alarm error: "HCX Interconnect Service Mesh Tunnel State Change"
• Service Mesh NE appliances are receiving an alarm error: "HCX Interconnect Service Mesh NE Tunnel State Change"
• In the HCX Manager UI -> Interconnect -> Service Mesh -> Appliances Tunnel status is "Down" for 1 or all of the Fleet Appliances (IX or NE) 

 

Error Details:

From HCX Manager -> Administration -> Alerts, the status reports: 

“Component Type: VirtualMachine

Message: Appliance (<Fleet_Appliance_Name>) status changed from unknown to down. Overall transport tunnel status is down. Overall encryption tunnel status is down. Service pipeline status is down.

InstanceID: <ID>”

 

From HCX Manager -> Interconnect -> Service Mesh -> View Appliances. Expand the appliance with status DOWN.

Sometimes, the tunnel may resume connectivity with a status of UP/Degraded. However, if it remains DOWN/Degraded, you can initiate "Run Diagnostics" to pinpoint the exact location of the traffic interruption. To execute Service Mesh Diagnostics, go to Interconnect -> Service Mesh -> Run Diagnostics.

Environment

VMware HCX

Cause

The "Overall transport tunnel status is down" error indicates an underlay network issue. Specifically, the uplink IP used for transport (UDP port 4500) is unable to communicate between the Fleet Appliances. 

Resolution

To troubleshoot the error related to HCX Service Mesh IX/NE appliances, you can follow these steps:

1. SSH to HCX Manager, access ccli, list appliances, navigate to logs, and search for errors:
   
   - ssh admin@hcx-manager-ip

   - Once connected, access ccli and list appliances
   ccli
list
   
   - Identify the IX/NE appliance (replace # with the specific appliance identifier)
   go #
   
   - SSH to appliance
   ssh
   
   -Grep for "DOWN" in messages
   grep DOWN /var/log/messages
   
   # Navigate to log directory and search for 'Traceroute' in messages log cd /var/log/
less messages (Find String "Traceroute" you can do this by using / or ?).
   
   Use the following command to test port connectivity
   curl -kv telnet://<IP address>:<port number>

2. Analyze the logs to pinpoint where the traffic flow is obstructed

   • Inside the less command, use N to move to the next occurrence and review entries sequentially.
   • Identify where the traffic flow is obstructed
   • Note any specific IPs, timestamps, or error messages indicating the problem area as shown in the screenshot.
     
     
     
     
     
     
     
     
     

3. Further troubleshooting based on findings:

   • Investigate network connectivity/underlay network to identify where the traffic is being halt.
   • Review firewall settings and HCX configuration for any misconfigurations.
   • Consult HCX documentation or support for specific error codes or troubleshooting steps.

 

Additional Information

For information on the required ports, access VMware Ports and Protocols and Network Diagrams for VMware HCX.

Ref: VMware HCX Documentation