NSX-T tier-0 logical router in an A/A topology, Internal BGP (iBGP) session are down between the service routers
search cancel

NSX-T tier-0 logical router in an A/A topology, Internal BGP (iBGP) session are down between the service routers

book

Article ID: 318313

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:
  • You recently upgraded to NSX-T 3.1, 3.1.1 or 3.1.2 and bgp_down alarms are generated indicating that BGP is down between 2 edge nodes in an active-active (A/A) cluster. 
  • These alarms may also trigger when an edge node is replaced in an edge cluster: "All BGP/BFD sessions are down".
  • With a tier-0 logical router in an A/A topology, there is an inter Service Router (Inter-SR) iBGP routing feature to handle asymmetric routing failures. It is noticed that this Inter-SR iBGP session(s) never get established.
  • From the tier-0 SR context on an NSX-T Data Center edge node, you can ping the iBGP peer IP address, but there may be packets lost.


Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

There is an issue with Inter-SR routing ports in the internal Virtual Routing and Forwarding (VRF) context which causes two edge nodes have the same MAC address.

In the example below the same MAC address '02:50:56:56:52:02' is being applied to the Inter-SR interfaces for both NSX-T Data Center edge nodes 2 and 3 in a three edge node cluster:

edge1(tier0_sr)> get neighbor
Wed May 12 2021 UTC 13:05:39.304
Logical Router
UUID : ee98c58b-599e-4e74-b506-fcd33e7d4f60
VRF : 9
LR-ID : 3082
Name : SR-Provider-Tier0
Type : SERVICE_ROUTER_TIER0
Neighbor
    Interface : 0cf071e2-537d-415c-a16e-f05d1e6d077a
    IP : 10.10.10.1
    MAC : b4:0c:25:e0:40:11
    State : reach
    Timeout : 341

    Interface : 79bfba70-7d14-4df5-bf6f-c5ad24e4103c
    IP : 169.254.0.131
    MAC : 02:50:56:56:52:02
    State : reach
    Timeout : 860

    Interface : 0cf071e2-537d-415c-a16e-f05d1e6d077a
    IP : 10.10.10.162
    MAC : 00:50:56:b4:53:8c
    State : reach
    Timeout : 938

    Interface : 0cf071e2-537d-415c-a16e-f05d1e6d077a
    IP : 10.10.10.2
    MAC : b4:0c:25:e0:80:11
    State : reach
    Timeout : 599

    Interface : 0cf071e2-537d-415c-a16e-f05d1e6d077a
    IP : 10.10.10.161
    MAC : 00:50:56:b4:c6:a4
    State : reach
    Timeout : 1031

    Interface : 79bfba70-7d14-4df5-bf6f-c5ad24e4103c
    IP : 169.254.0.132
    MAC : 02:50:56:56:52:02
    State : reach
    Timeout : 577


Note: This issue can occur on a two node Edge cluster as well.
This issue affects all versions of NSX-T 3.x prior to NSX-T 3.1.3.

Resolution

Issue is resolved in NSX-T 3.1.3.

Workaround:
The following steps can also be followed. 

1. Turn off the Inter-SR iBGP option from the NSX Manager UI. This will delete all internal routing ports and iBGP sessions.
Networking -> Tier-0 Gateways -> T0 Edit-> BGP -> Turn Off 'Inter SR iBGP':

image.png

2. Turn on the Inter-SR iBGP option again, which will create new internal routing ports without the duplicate MAC addresses, allowing the iBGP sessions to successfully establish.

Additional Information

Impact/Risks:
Inter SR iBGP session will not work and datapath will be impacted for asymmetric and ECMP topologies.