HCX Service Mesh Tunnels DOWN due to IPsec SA Conflict with Old HCX Environment

Products

• VMware HCX

Issue/Introduction

• Standard transport validation via foutrace completes successfully.
• MTU settings are verified as correct.
• There was another HCX environment which was used previously with the same IP addresses.
• IPsec logs (/var/log/messages) show secondary tunnels stuck in a CONNECTING state during the IKE_AUTH phase:
  

  [Info-configer] : ipsec status: Security Associations (0 up, 3 connecting):
     (unnamed)[3]: CONNECTING, 192.0.2.60[%any]...192.0.2.55[%any]
     (unnamed)[2]: CONNECTING, 192.0.2.59[%any]...192.0.2.54[%any]
     (unnamed)[1]: CONNECTING, 192.0.2.58[%any]...192.0.2.53[%any]

• In /var/log/system_events the IP SEC tunnel is not established, instead IPIP_FOU_DYNAMIC Tunnel is up:

  {"IPIP_FOU Tunnel is up","metadata":{"tunnelId":"te_7","tunnelType":"IPIP_FOU"}}
  {"IPIP_FOU Tunnel is up","metadata":{"tunnelId":"te_0","tunnelType":"IPIP_FOU"}}
  {"IPIP_FOU Tunnel is up","metadata":{"tunnelId":"te_1","tunnelType":"IPIP_FOU"}}
  {"IPIP_FOU_DYNAMIC Tunnel is up","metadata":{"tunnelId":"d_232","tunnelType":"IPIP_FOU_DYNAMIC"}}
  {"IPIP_FOU_DYNAMIC Tunnel is up","metadata":{"tunnelId":"d_230","tunnelType":"IPIP_FOU_DYNAMIC"}}
  {"IPIP_FOU_DYNAMIC Tunnel is up","metadata":{"tunnelId":"d_231","tunnelType":"IPIP_FOU_DYNAMIC"}}
  {"IPIP_FOU_DYNAMIC Tunnel is up","metadata":{"tunnelId":"d_233","tunnelType":"IPIP_FOU_DYNAMIC"}}

 

Cause

This issue occurs due to an IP address conflict with an older HCX environment.

When a new Cloud Service Mesh is deployed using the exact same IP addresses as a previous deployment, any undecommissioned appliances from the old HCX environment will continue attempting to establish secure connections to those destination IPs. As a result, the newly deployed Cloud IX and NE appliances receive simultaneous encryption connection requests from two different source IP addresses (the new on-premises appliances and the old HCX environment appliances). The IPsec daemon cannot handle conflicting connection attempts targeting the same local listener IP, causing the secure tunnels to fail and remain stuck in a CONNECTING state.

Resolution

To resolve this issue, you must cleanly decommission the old HCX environment Service Mesh appliances.

1. Log in to the old on-premises HCX Manager.
2. Navigate to the Interconnect interface.
3. Remove the stale Service Mesh.

This action will power off and remove the old IX and NE appliances, terminating the conflicting UDP/IKE traffic and allowing the new IPsec SA negotiations to complete successfully.

 

Additional Information

 Destination NAT configurations are not supported.
IX appliance tunnel DOWN with error "Service Pipeline is Down"
HCX Tunnel down with Error "Overall encryption tunnel status is down. Service pipeline status is down."