VMware NSX Generates BGP down alarms for Tier 0 Logical Router configured with Route based VPN

Products

• VMware NSX

Issue/Introduction

Symptoms:

• The NSX-T Tier 0 logical routers are in Active Standby mode.
• You have configured a route based VPN.
• On the edge node for the active Tier 0 instance, BGP shows up and established.

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

169.254.1.1                      64064       Estab 2d16h31m     NC  27878   24257   10     42
169.254.2.1                      64512       Estab 2d16h30m     NC  24234   24254   1      51
169.254.3.1                      64512       Estab 2d16h30m     NC  24234   24256   1      52
 

• On the edge node for the standby Tier 0 instance, BGP shows as idle.

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

169.254.1.1                       64064       Idle  never        NC  0       0       0      0
169.254.2.1                       64512       Idle  never        NC  0       0       0      0
169.254.3.1                       64512       Idle  never        NC  0       0       0      0

• The NSX Manager generates alarms that BGP is down for each neighbor on the standby node:

"summary" : "BGP neighbor down.",
    "description" : "In Router xxxxxxxx-b860-44fd-a3f6-xxxxxxxxxxxx, BGP neighbor xxxxxxxx-cafa-4e5e-ac69-xxxxxxxxxxxx(9000::9001:2) is down.",
    "recommended_action" : "1. Invoke the NSX CLI command `get logical-routers`. 2. Switch to service-router xxxxxxxx-14d6-484d-9494-xxxxxxxxxxxx. If the reason indicates Network or config error - 3. Invoke the NSX CLI command `get bgp neighbor summary` to check the BGP neighbor status. If the reason indicates `Edge is not ready`, check why the Edge node is not in good state. 4. Invoke the NSX CLI command `get edge-cluster status` to check reason why Edge node might be down. 5. Invoke the NSX CLI commands `get bfd-config` and `get bfd-sessions` to check if BFD is running well. 6. Check any Edge health related alarms to get more information. Check /var/log/syslog to see if there are any errors related to BGP connectivity.",

Environment

VMware NSX-T

Cause

On the Standby Tier 0 logical router, we expect to see BGP down, when the Tier 0 logical router is in A/S mode and the neighbor is over an IPSEC VPN (VTI Interface), as the VPN is not active on this edge node.
If we do get interfaces, we see the VTI (Virtual Tunnel Interface) interface which is connecting to the BGP neighbor, is DOWN, which is expected when the logical router is in standby mode:

{
                "ifuuid": "xxxxxxxx-ea54-49be-8237-xxxxxxxxxxxx",
                "ifuid": 538,
                "type": "vti",
                "ptype": "vti",
                "enable-firewall": true,
                "enable-firewall-pbr": false,
                "lrouter": "xxxxxxxx-14d6-484d-9494-xxxxxxxxxxxx",
                "admin": "up",
                "internal_operation": "down",
                "urpf-mode": "PORT_CHECK",
                "policy uuid": "00000000-0000-0000-0000-000000000000",
                "ipns": [
                    "169.254.1.2"
                ],

Therefore if the VTI interface is down, the BGP session will be down.
As it is the Standby Tier 0 logical router, we expect IPSEC VPN not to be up on Standby Tier 0 logical router and therefore do not generate a IPSEC VPN alarm, this issue here is that BGP still checks and generates an alarm.

Resolution

This issue is resolved in VMware NSX 4.0.0, available at VMware downloads.

Workaround:
This issue does not impact the dataplane, it displays an alert about BGP being down, which it is expected to be, no workaround is required.