NSX Edge Node iBGP Session Down between RTEP interfaces

Products

• VMware NSX
• VMware Cloud Foundation

Issue/Introduction

VM-to-VM communication across Local Manager (LM) sites over a shared Global Logical Switch fails. Inter-site traffic between hosts is disrupted, specifically affecting inter-site Service Router (SR) internal BGP (iBGP) sessions between NSX Edge Node Remote Tunnel Endpoints (RTEPs).

• Error Symptoms: Inter-SR iBGP between NSX Edge Node RTEPs shows DOWN in the NSX UI or not in an established state from CLI.

  • NSX Edge Node admin CLI check:

    • edge> set debug

    • edge> vrf internal

    • edge> get bgp neighbor summary

  • NSX UI check:

    • Log into NSX Global Manager UI with admin credentials

    • Navigate to Networking > Tier-0 Gateways

    • Locate the Global Tier-0 Gateway in question and click the menu icon (three vertical dots) next to its name, then select Generate BGP Summary.

    • Select the System Configured Neighbors tab and check the Connection Status column

• Packet Capture Analysis: Packet captures executed on the ESXi hosts where the NSX Edge Nodes reside show ICMP requests (from 169.254.32.# to 169.254.32.#) resulting only in outbound ARP requests leaving the ESXi hosts, with no ARP replies returned to the Edge Node VMs.

  • Packet Captures for VMware NSX.

Environment

• Product: VMware NSX / VMware Cloud Foundation (VCF)

• Component: NSX Edge Node / Federation / Stretched Networking / RTEP

Cause

The Local Tunnel Endpoint (TEP) VLAN and the Remote Tunnel Endpoint (RTEP) VLAN were configured using the same VLAN ID on one site. Sharing the same VLAN ID prevents Layer 3 routing and encapsulation handling for inter-site traffic across ESXi hosts, blocking ARP resolution for internal link-local IP addresses (169.254.32.#) and preventing iBGP neighbor establishment.

Resolution

• Provision a new, dedicated VLAN ID for the RTEP interfaces on the physical network switches.

• Configure the new RTEP VLAN across all physical switches in the transport path.

• Ensure physical switch trunk ports connected to the ESXi host uplinks carry the newly provisioned RTEP VLAN tag.

• Reconfigure the NSX Edge Node RTEP interface settings to utilize the new dedicated RTEP VLAN (Configure Edge Nodes for Stretched Networking).

• Verify that ARP requests resolve successfully between site RTEP endpoints (169.254.32.#) and confirm inter-SR iBGP sessions transition to the ESTABLISHED state.

Additional Information

Configure Edge Nodes for Stretched Networking