ARP fails in L2VPN envrionment with Standard vSwitch and independent uplinks

Products

• VMware NSX

Issue/Introduction

Symptoms:

Two Sites (A and B) are connected by a L2VPN running on nsx edge gateways; in site B a Vlan is stretched across Standard vSwitch and Portgroups to Physical Switches, the edge gateway is deployed on a esx host which is connected to the Physical Switches with multiple independent uplinks. Workloads residing in Site A cannot communicate with Workloads placed on the physical Network in Site B. Investigations shows that arp requests are sent from site A across the L2VPN to site B, but no arp replies are received. In the log of the EdgeGateway mac-flapping between a vNic Interface and a vpn interface (na1, tap1) is reported.

Environment

VMware NSX for vSphere 6.2.x

Cause

The arp requests are sent from site A to site B through the vpn tunnel. The site B l2vpn edge receives the broadcast frame on its vpn interface and creates a source-mac to vpn-port mapping in its mac-table, the esx host running the edge forwards it out to the physical Switch via its first Uplink vmnic. The physical switch receives the broadcast frame and sends it onto all relevant ports for this Vlan with the exception of the Port where the frame was received. As the Host is connected to the Switches with multiple Uplinks, the broadcast frame reaches the host and the edge a second time, now from the trunk interface where the physical switches are connected to. This overwrites the first entry in the edge mac/port mapping table for the arp request from the site A system, now it appears as being received from the local network. The corresponding arp reply is therefore not forwarded across the l2vpn to site A.

Resolution

To resolve this issue, enable PromiscousMode on the portgroup security policy on site B( where the Physical Switches are connected to the edge by a vSS), to create the sink port for the edge. When this is enabled, looped packets forward to the edge from the trunk interface.

 

To work around this issue, activate RPF check for the Promiscous Mode from the esx rootshell:

 

esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1

esxcli system settings advanced list -o /Net/ReversePathFwdCheckPromisc

Path: /Net/ReversePathFwdCheckPromisc

Type: integer

Int Value: 1

Default Int Value: 0

Min Value: 0

Max Value: 1

String Value:

Default String Value:

Valid Characters:

Description: Block duplicate packet in a teamed environment when the virtual switch is set to Promiscuous mode.

 

• In PortGroup security policy, set PromiscousMode from Accept to Reject and back to Accept to activate the configured change.