Total Layer 2 Bridging Outage During Upstream Switch Maintenance Due to Symmetrical Edge Uplink Cabling

Products

• VMware NSX

Issue/Introduction

• Occurs within an NSX Bare Metal Edge Cluster utilizing Layer 2 VLAN Bridging and Active/Standby Edge HA Profiles.
• A single upstream physical Top-of-Rack (ToR) switch is taken offline, either for a planned maintenance window or due to an unexpected hardware failure.
• A complete Layer 2 bridging outage is experienced across the environment.
• Node-Level High Availability (HA) failover initiates but fails to restore datapath traffic.

Environment

VMware NSX

Cause

This issue is caused by a physical architecture and uplink mapping design flaw, specifically symmetrical physical cabling to the upstream switches.

If both Edge nodes in the HA cluster are cabled identically to the upstream physical network (e.g., uplink1 on both Edge A and Edge B physically connect to Switch A, and uplink2 on both nodes connect to Switch B), taking down Switch A severs the active datapath across the entire cluster simultaneously.

Because the active path is lost on both nodes at the exact same moment, a Node-Level HA failover is physically impossible; the Standby node cannot restore traffic because it has also lost its corresponding physical uplink.

Reviewing the datapath logs (syslog) during the switch maintenance reveals that both Edge nodes detect a physical link-down event on their primary interfaces at the exact same millisecond.

Edge-A
2###-##-##T##:##:01.123Z Edge-## ... dataplane: [INFO] Link state DOWN for interface fp-eth1 (uplink1)
2###-##-##T##:##:01.145Z Edge-## ... dp-rc: [WARN] BFD session down for tunnel ... reason: Physical link down
2###-##-##T##:##:01.147Z Edge-## ... bridge: [WARN] Datapath connectivity lost on Active node

Edge-B
2###-##-##T##:##:01.123Z Edge-## ... dataplane: [INFO] Link state DOWN for interface fp-eth1 (uplink1)
2###-##-##T##:##:01.148Z Edge-## ... dp-rc: [WARN] BFD session down for tunnel ... reason: Physical link down
2###-##-##T##:##:01.150Z Edge-## ... bridge: [WARN] HA Failover triggered, but active uplink unavailable

When this happens because both the Active and Standby NSX Edge nodes simultaneously lose connectivity on their active forwarding paths.

Resolution

To resolve this issue and ensure HA failover can function during a single-switch maintenance window, the physical paths must be staggered so that the Active and Standby nodes do not rely on the same physical switch for the same logical uplink.

This can be achieved via either a physical or logical fix:

Option 1: Physical Datacenter Recabling

• Physically cross-cable the Edge interfaces to the upstream ToR switches to stagger the fault domains.
• Edge A: Connect Bridge-Port 1 to Switch A, Bridge-Port 2 to Switch B.
• Edge B: Connect Bridge-Port 1 to Switch B, Bridge-Port 2 to Switch A.

Option 1: Logical Software Workaround

If a physical recabling the datacenter isn't possible, you can logically "cross" the cables by remapping the uplink profiles specifically on the impacted Edge node

1. Log into the NSX Manager UI.
2. Edit the Edge Node configuration for the impacted node.
3. Modify the physical NIC mapping so that it is inverted compared to the Primary node:
   • Map uplink1 to fp-eth3 (or the secondary physical interface).
   • Map uplink2 to fp-eth1 (or the primary physical interface).
4. Save the configuration. This ensures that if Switch A goes down, Edge B uplink1 remains active via Switch B and vice versa.

Additional Information

BME Edge Uplink Profile

NSX-T Bare Metal Edge bridge does not support multiple VLAN uplinks

VMware NSX Reference Design Guide