BFD tunnels between Bare Metal Edges and Hosts go Down and continue to flap when BME TEPs are hit with scanning traffic

Products

• VMware NSX

Issue/Introduction

Symptoms:

- TEP tunnels between Bare Metal Edges and Host Transport nodes go Down and flap continuously
- Frequent MAC changes are observed for TEP IPs in physical switches which are updating MAC tables by monitoring traffic (ex: Cisco ACI)
- Port scanning software is sending ICMP, UDP, or TCP traffic to a Bare Metal Edge with multiple TEP interfaces 
- Captures on Edge TEP interfaces during ping scanning show return ICMP traffic is leaving the incorrect TEP interface
For example:
If TEP A is pinged, an ICMP reply egresses TEP A with TEP A's source IP and MAC. 
If TEP B is pinged, an ICMP reply is also sent out TEP A, with TEP B's source IP and TEP A's source MAC.
 
- Simultaneous captures at time of flap at Edge TEP interface and ESXi uplink show BFD Control packets not reaching the Bare Metal Edge even though they leave the ESXi host.

Environment

VMware NSX-T Data Center

Cause

Return TCP traffic having an incorrect MAC and IP pair as described above can poison the MAC table of the upstream physical switch.
With the incorrect MAC binding in the upstream physical switch, BFD packets are dropped, TEP tunnels are marked as Down, and this can cause dataplane impact.

Resolution

This issue is resolved in NSX-T  3.1.2.3, 3.1.3.3, 3.1.4.0, 3.2.0.0, and later releases.

Workaround:
Stop port scanning traffic towards the Bare Metal Edge TEP interfaces.

Additional Information

Impact/Risks:
Dataplane traffic that would pass through the overlay where TEP tunnels are down is impacted.