MAC Flapping observed in an NSX Environment

Products

• VMware NSX

Issue/Introduction

Symptoms:
In an environment, where ESXi management interface vmk0 and VXLAN VTEPs interfaces share the same vDS and NICs:

• MAC flapping alerts may be observed on the physical switch.
• The MAC flapping is the MAC address of vmk0.

Cause

By default at ESXi installation time, the management vmkernel vmk0 inherits the MAC address of vmnic0.

In an NSX environment configured for unicast and hybrid mode, BUM traffic between different VTEP segments is sent directly to a UTEP host. The UTEP host is responsible for replicating the traffic locally to each host in its VTEP segment. When this traffic is replicated, the source MAC of the packets is set to be the MAC of the vmnic it was received on.

MAC Flapping will occur if the following 2 conditions are met:

• vmk0 is active on a NIC other than vmnic0 
• VTEP interface is active on vmnic0

If vmk0 is active on a NIC other than vmnic0, it will send its traffic with source MAC of vmnic0 through another NIC vmnicX, this in itself causes no issues. The VTEP vmkernel is active on vmnic0.
When the VTEP receives BUM traffic for replication, it will send it out with source MAC of vmnic0.

The physical switch therefore sees traffic with the same source MAC address (vmnic0 MAC) coming from vmnic0 and vmnicX and MAC flapping occurs.

Resolution

This is a known issue affecting NSX for vSphere 6.3.x and NSX for vSphere 6.4.x.

Currently, there is no resolution.

Workaround:
To work around this issue, ensure that vmk0 is actively using vmnic0 or VXLAN VTEP is not using vmnic0.