Tier0 removing already installed BGP routes in certain ECMP scenario

Products

• VMware NSX

Issue/Introduction

This article provides information on the non-standard BGP route selection algorithm in an ECMP corner case.

Symptoms:
During BGP best path computation, when number of best paths available are more than the 8 maximum paths allowed on Edge Node you experience this symptom:

BGP may delete an already selected path from earlier computation and give preference to a new path even though they both are equally qualified as best paths.

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Cause

For this explanation, consider that the maximum paths allowed are 3, even if on the Edge Node the maximum is in reality 8.

1. Edge node received a BGP route for prefix a.b.c.d from a neighbor node. Neighbor node has router-id as 10.1.1.103. Edge Node installs this route into Data Plane.
    
2. Edge node received another BGP route for the same prefix a.b.c.d from another neighbor node. This neighbor node has router-id as 10.1.1.102 with all attributes like weight, local preference, MED are exactly matching with the earlier selected path. Edge Node installs this route into Data Plane.
   
   a. Edge Node installs this route and existing “old route” (from node 10.1.1.103) is not disturbed.
    
3. Edge node received another BGP route for prefix a.b.c.d from another neighbor node. This neighbor node has router-id as as 10.1.1.104 with attributes like weight, local preference, MED matching are exactly matching with earlier two selected paths. Edge Node installs this route into Data Plane.
   
   a. Edge Node installs this route and existing “old routes” are not disturbed.
    
4. Even though the BGP routes received in the above two cases are from neighbors with lower router-id and higher router-id, no change to existing/old routes and they were kept intact in data plane.
    
5. Edge node received another (fourth) BGP route for prefix a.b.c.d from neighbor node with router-id as 10.1.1.101. For example, now ECMP paths exceeds maximum paths allowed (3).
   
   a. Edge node does not disturb the first ever BGP route installed for this prefix even though the new route is received from lower router-id neighbor. For example: From 10.1.1.101. So, the BGP route received from neighbor with router-id 10.1.1.103 will still be present in FIB.
   
   b. However, now lower router-id check is performed among the remaining BGP routes from 10.1.1.102, 10.1.1.104 and 10.1.1.101. As per this check, FRR selects the remaining two BGP routes from 10.1.1.102 and 10.1.1.101. So the existing BGP route from 10.1.1.104 will be removed from Data Plane and a new BGP route from 10.1.1.101 will be installed into Data Plane.

Resolution

To resolve this issue, avoid having more then 8 best paths for any given network. You could have more than 8 paths, but make sure in this case that the additional paths have a BGP attribute to make them not preferred