HCX DR VM protection fails with following error "Interconnect appliance is not deployed"

Products

• VMware HCX

Issue/Introduction

• Below is the error message encountered stating, "Interconnect appliance is not deployed - Interconnect appliance is not deployed" after a VM is selected for HCX DR protection. 
  
• HCX IX appliance in the service mesh (Interconnect -> Service Mesh -> Appliances) shows tunnel status as 'Down'.
  

Environment

VMware HCX

Cause

The "Overall transport tunnel status is down" error indicates an underlay network issue. Specifically, the uplink IP used for transport (UDP port 4500) is unable to communicate between the Fleet Appliances. 

Resolution

To troubleshoot the error related to HCX Service Mesh IX/NE appliances, you can follow these steps:

1. SSH to HCX Manager, access ccli, list appliances, navigate to logs, and search for errors:
   
   - ssh admin@hcx-manager-ip

   - Once connected, access ccli and list appliances
   ccli
list
   
   - Identify the IX/NE appliance (replace # with the specific appliance identifier): go #
   
   - SSH to appliance by typing: ssh

2. Analyze the logs to pinpoint where the traffic flow is obstructed

   • Grep for "DOWN" in messages: grep DOWN /var/log/messages
   • Identify where the traffic flow is obstructed
   • Note any specific IPs, timestamps, or error messages indicating the problem area as shown in the screenshot.
     
     
     # Navigate to log directory and search for 'Traceroute' in messages log cd /var/log/
less messages (Find String "Traceroute" you can do this by using / or ?).
     
     
     

3. Further troubleshooting based on findings:

   • Investigate network connectivity/underlay network to identify where the traffic is being halt.
   • Review firewall settings and HCX configuration for any misconfigurations.
   • Consult HCX documentation or support for specific error codes or troubleshooting steps.

 

Additional Information

• For information on the required ports, access VMware Ports and Protocols and Network Diagrams for VMware HCX.
• Ref: VMware HCX Documentation 
• There is a known issue with the Palo alto firewalls where UDP IPsec session gets blocked, refer here for a workaround: