NSX Edges establish BGP Peering via all local IP addresses, but after a short time, one or more peers lose connectivity.

Products

• VMware NSX

Issue/Introduction

• NSX Edges have at least 2 IP addresses each, 1 per uplink.
• Each Edge uplink IP address should be accessed via a different VLAN (i.e. VLAN 100 and VLAN 200) and a different uplink.
• BGP will establish to all peers across all uplinks for a short time
• After BGP peering is established, 1 or more BGP pair will lose peering from the Edge uplink. (i.e. VLAN 100 peers will remain up, but VLAN 200 peers will go into Connect state)
• A packet capture will show traffic meant to exit the edge on Uplink 2 tagged for VLAN 200 is instead leaving on Uplink 1 and tagged for VLAN 100.

Example IP scheme:

• Edge 1 Uplink 1 has IP A.A.A.2/26 and peers with Physical Router 1 using IP A.A.A.1/26 over VLAN 100.
• Edge 1 Uplink 2 has IP B.B.B.2/26 and peers with Physical Router 2 using IP B.B.B.1/26 over VLAN 200.
  
  
• Edge 2 Uplink 1 has IP A.A.A.3/26 and peers with Physical Router 1 using IP A.A.A.1/26 over VLAN 100.
• Edge 2 Uplink 2 has IP B.B.B.3/26 and peers with Physical Router 2 using IP B.B.B.1/26 over VLAN 200.

Environment

• VMware NSX
• VCF

Cause

Prior to establishing BGP peering, traffic to and from B.B.B.1/26 on Uplink 2 on each Edge traverses VLAN 200 as expected. After initial peering takes place, both Edges receive route updates. Traffic to B.B.B.1 is now sent out of Edge’s uplink 1 using VLAN 100. 

This is caused by a misconfiguration at A.A.A.1 that informs the Edge there is a more specific route to B.B.B.1/32 accessible via VLAN 100 and peer A.A.A.1. This new route causes the Edge to redirect all BGP traffic to B.B.B.1/26 via Uplink 1/VLAN 100 instead of Uplink 2/VLAN 200.

A /32 route is more specific than a /26 route, and therefore takes precedence over /26, even if it the /26 is a directly connected route on an uplink.

Resolution

Ensure Physical Router 1 is not advertising incorrect routes to the Edge.