Routes learned via inter SR routing (Edges deployed in Active/Active mode) are advertised upstream via eBGP during an upgrade from 2.4.x/2.5.x to 3.x

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• During an upgrade to NSX-T 3.x, network connectivity to the virtual machines hosted by the NSX-T domain is impacted.
• Prefixes learned from a top of rack switch are re-advertised to other top of rack switches in different BGP autonomous system.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.5.x

Cause

This issue only affects 2.x to 3.x upgrades

Customers upgrading from version prior to 3.0 to any version 3.0 and higher may see inter-SR routes being advertised to the configured BGP neighbors on an Active-Active T0 Gateway when BGP filters or route-maps are not used for outbound advertising. This issue will be seen transiently on the edges of versions older than 3.x when the upgrade process is in progress and some edges are upgraded to version 3.x.

Customers upgrading from NSX-T version prior to 3.0 to any version 3.0 and higher may see northbound prefixes being advertised back to the physical networking fabric under the following circumstances:

• Tier-0 gateway is deployed in an Active/Active topology
• Inter-SR iBGP is enabled in NSX-T Manager.
• Route-map or prefix-list are not used on the Tier-0 gateways to filter NLRI to the BGP neighbors in the physical fabric.
• Route-map or prefix-list are not used on the physical network fabric to filter NLRI learned from the Tier-0 gateways.
• Top of Rack switches are in a different BGP autonomous system between each other.

1.3.1. Topology used

The following diagram represents an edge cluster composed with 2 edge nodes during an NSX-T upgrade, edge node "EN1" is running NSX-T version 3.x and edge node "EN2" is running NSX-T version 2.5.x

A Tier-0 gateway is deployed in Active/Active mode and eBGP is configured towards two BGP neighbors that are in different BGP Autonomous System from each others (65100 for ToR-01 and 65200 for ToR-02). The Tier-0 gateway is configured with BGP Autonomous System 65000. There is no BGP filtering used in this topology between the physical fabric and the NSX-T Tier-0 gateways.

Step 1: As demonstrated in the figure below, ToR-01 is advertising an NLRI about network 1.1.1.0/24 towards the Tier-0 SR 01.

Step 2: As Inter SR iBGP is enabled, the Tier-0 SR 01 advertises that NLRI towards the Tier-0 SR 02. This prefix is installed in the routing table of the Tier-0 SR 02.

 

Step 3:  Since there is no route-maps or prefix-lists configured on both the Tier-0 gateway and the top of rack switches, NLRI are not filtered and 1.1.1.0/24 will be advertised towards the networking fabric. The ToR-02 is in a different BGP autonomous system than ToR-01 so the BGP Update for that prefix with the AS_PATH "65000-65200" will be accepted. ToR-02 will accept that BGP update and based on the physical network topology in place can prefer the path through the Tier-0 gateway who will act as a transit gateway. If the top of rack switches are configured with the BGP "Allow-AS-In" feature, the BGP update for that prefix will also be accepted.

 

 

Resolution

The issue is not present in NSX-T 3.x releases 

For more information on NSX-T Inter-SR BGP Peering, please refer to the NSX-T design guide.

Workaround:

Workaround

Option 1: 

Use UI to disable inter SR routing under the Tier-0 BGP configuration

1. Before starting the upgrade, disable inter_SR routing on NSX-T Tier-0 gateways.
2. Upgrade the NSX-T edges to 3.x or 3.x.x (depending the upgrade path).
3. At the end of the upgrade, enable inter_SR routing on the NSX-T Tier-0 gateways.

Option 2:

1. Filter NLRI updates sent from the Tier-0 gateways on the top of rack switches using route-maps, prefix-list, as-path access-list, communities.
   1. example: Filter routes not originated by the NSX-T Tier-0 autonomous system.

Option 2 is applicable to 3rd party routers with configuration changes.

VMware recommends disabling the inter-SR iBGP peering feature before upgrading the NSX-T edge to the 3.x version.