Intermittent Connectivity on NSX Segment with Edge L2 Bridging NSX

Products

• VMware NSX

Issue/Introduction

• Intermittent ping failures between VMs on the same overlay segment.
  
  
• East-West communication is impacted only when workloads are on different ESXi hosts.
  
  
• NSX Traceflow shows packets dropped at the "Logical Switch on Edge node."
  
  
• Connectivity may succeed briefly after vMotion but fails once MAC learning tables update.
  
  
• Packet captures show Geneve encapsulated packets incorrectly forwarded to the Edge TEP instead of the destination host TEP.

Environment

• VMware NSX-T 3.x
• VMware NSX 4.x
• ESXi 7.x and 8.x
• NSX Edge L2 Bridge configured using a VDS Portgroup with Promiscuous Mode.

Cause

This issue is caused by the vSphere Distributed Switch (VDS) port security policy or Reverse Path Forwarding (RPF) checks on the ESXi host. When L2 bridging is active on a VDS portgroup, the Edge VM forwards traffic for multiple overlay MAC addresses.

If ReversePathFwdCheckPromisc is not enabled, the host may drop this egress traffic or cause MAC learning inconsistencies, leading to packets being looped or dropped by the security layer.

Resolution

To ensure stable L2 bridging when the Edge VM is connected to a VDS portgroup, perform the following steps on every ESXi host that may host the Edge Transport Nodes:

1. Log into the ESXi host via SSH as root.
   
   
2. Enable the advanced host setting for Reverse Path Forwarding on promiscuous ports:
   esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1
   
   
3. Verify the setting is applied:
   esxcli system settings advanced list -o /Net/ReversePathFwdCheckPromisc
   
   
4. Navigate to the vSphere Client and locate the VDS Portgroup used for the Edge Bridge.
   
   
5. Edit the Portgroup settings and navigate to Security.
   
   
6. Set Promiscuous Mode to Reject, then click OK.
   
   
7. Edit the settings again, and set Promiscuous Mode back to Accept to initialize the new host setting.
   
   
8. Ensure Forged Transmits is set to Accept.

Additional Information

The preferred configuration for L2 bridging is Option 3: Edge VM is connected to an NSX segment, which leverages native MAC Learning and does not require Promiscuous Mode.

Reference Documentation:

• Configure an Edge VM for Bridging in Manager Mode
• Duplicate Multicast or Broadcast Packets are Received by a Virtual Machine
• How promiscuous mode works at the virtual switch and portgroup levels

For further assistance, please see Contact Broadcom Support. Scroll to the bottom of the page and click on your respective region.