Intermittent Connectivity and Physical Port Shutdowns on NSX Segment with Edge L2 Bridging NSX

Products

• VMware NSX

Issue/Introduction

• Intermittent ping failures between VMs on overlay segments.
• East-West communication is impacted only when workloads are on different ESXi hosts.
• Network connectivity drops coincide with the physical switch port shutting down (Err-Disabled state) due to BPDU Guard triggering.
• NSX Traceflow shows packets dropped at the "Logical Switch on Edge node."
• Connectivity may succeed briefly after vMotion but fails once MAC learning tables update.
• Packet captures show Geneve encapsulated packets incorrectly forwarded to the Edge TEP instead of the destination host TEP.

Environment

• VMware NSX-T 3.x
• VMware NSX 4.x
• ESXi 7.x and 8.x
• NSX Edge L2 Bridge configured using a VDS Portgroup with Promiscuous Mode.

Cause

This issue is caused by a combination of vSphere Distributed Switch (VDS) port security policies and physical switch loop protection mechanisms:

1. Host Level: This issue is caused by the vSphere Distributed Switch (VDS) port security policy or Reverse Path Forwarding (RPF) checks on the ESXi host. When L2 bridging is active on a VDS portgroup, the Edge VM forwards traffic for multiple overlay MAC addresses. If ReversePathFwdCheckPromisc is not enabled, the host may drop this egress traffic or cause MAC learning inconsistencies, leading to packets being looped or dropped by the security layer.

2. Physical Switch-Level: Because the NSX Edge node performs L2 bridging, it may forward Spanning Tree (BPDU) frames. Physical switch ports connected to the Edge host often have BPDU Guard enabled. When the switch detects these BPDUs, it interprets the bridging activity as a loop hazard and shuts down the port (Err-Disabled).

Resolution

To ensure stable L2 bridging when the Edge VM is connected to a VDS portgroup, perform the following steps on every ESXi host that may host the Edge Transport Nodes:

1. Log in to the ESXi host via SSH as root.
2. Enable the advanced host setting for Reverse Path Forwarding on promiscuous ports:

   esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1

3. Verify the setting is applied:

   esxcli system settings advanced list -o /Net/ReversePathFwdCheckPromisc

4. Navigate to the vSphere Client and locate the VDS Portgroup used for the Edge Bridge.
5. Edit the Portgroup settings and navigate to Security.
6. Set Promiscuous Mode to Reject, then click OK.
7. Edit the settings again, and set Promiscuous Mode back to Accept to initialize the new host setting.
8. Ensure Forged Transmits is set to Accept.

Also, make sure to configure the connected physical switch ports to ignore or filter BPDUs to prevent the physical switch from shutting down the ports.

Alternatively, removing the Bridge profile all together will resolve the issue. If Bridging is deemed unnecessary and routing can be configured in the physical network for communication between physical servers and VMs on NSX overlay segments, this behavior can be avoided. 

Additional Information

The preferred configuration for L2 bridging is Option 3: Edge VM is connected to an NSX segment, which leverages native MAC Learning and does not require Promiscuous Mode.

Reference Documentation:

• Configure an Edge VM for Bridging in Manager Mode
• Duplicate Multicast or Broadcast Packets are Received by a Virtual Machine
• How promiscuous mode works at the virtual switch and portgroup levels
• Common Edge Bridging

For further assistance, please see Contact Broadcom Support. Scroll to the bottom of the page and click on your respective region.