Tier-0 status down for 1 Edge node in an active/active cluster

Products

• VMware NSX

Issue/Introduction

• You have a Tier-0 gateway with 2 or more Edges that are active/active.
• BGP neighbor status is Established on one Edge node.
• BGP neighbor status is Connected on the second Edge node.
• You have an alarm for BGP neighborship down
• Ping performed from within the Tier-0 SR VRF to BGP peers fails on the second Edge node
  
  
• Example with 2 BGP peers:
  172.###.###.15
172.###.###.14
  

  Enter the VRF of the Tier-0 SR on the second Edge node:
  > get logical-routers
> vrf 1
> get route

  t0c> * 172.###.###.0/25 is directly connected, uplink-###, 03w0d00h
isr> * 172.###.###.15/32 [200/0] via 169.254.0.###, inter-sr-###, 03w0d00h
isr> * 172.###.###.14/32 [200/0] via 169.254.0.###, inter-sr-###, 03w0d00h

• We see that the /32 address of the BGP peers are being advertised to the Edge nodes by the Top-of-Rack switch and that the second Edge is learning the /32 route via the first Edge (inter-sr), even though there is a /25 advertised to the uplink that is directly connected.

  Note: The preceding entries are only examples. Make sure that there is a broader prefix being advertised that includes the BGP peers. In this example a /25.

Environment

VMware NSX

Cause

The BGP peer IP addresses are being advertised as /32 prefix routes.
In networking, a /32 is the most specific route possible and takes precedence over broader prefixes (such as the /25 subnet typically assigned to the uplink VLAN).
When iBGP is enabled between Edge nodes, Edge2 may learn the /32 route for the ToR neighbor via Edge1. Because the /32 is more preferred than the /25 advertised by the ToR on the local uplink, Edge2 attempts to reach the BGP peer by hair-pinning traffic through Edge1 instead of exiting its own physical interface.

Resolution

To resolve this issue, you must prevent the /32 BGP peer routes from being learned or preferred via the inter-sr VRF by applying a prefix list to deny these specific routes. This can be done on the ToR or on the Tier-0 Gateway. 

1. Log in to the NSX Manager UI.
2. Navigate to Networking > Tier-0 Gateways
3. Edit the affected Tier-0 Gateway.
4. Under BGP > BGP Neighboursm, verify if a Route Map with a Prefix List is already applied on the In Filter.
   1. If there isn't, under Routing, create a Prefix List that denies the specific /32 IP addresses of your BGP neighbors.
      
   2. If there is, edit the existing Prefix list to include the /32 networks for both BGP peers with action: Deny. Make sure the deny prefixes are above any allow as they are processed top to bottom.
5. Navigate to BGP > Neighbors.
6. Edit the BGP neighbor configuration and apply the newly created Prefix List as an Inbound Filter.
7. Verify that the /32 route is no longer preferred via iBGP and that both Edge nodes now establish BGP sessions directly via their respective local uplinks by checking get route in the Tier-0 SR VRF.
8. Verify BGP is established: get bgp neighbor summary

Additional Information

BGP neighborship is down alarm