- The issue occurs when physical environment (eg. Cisco ACI) uses data packets for IP discovery, rather than just arp snooping, and BFD is enabled within NSX.
- BFD is enabled on NSX environment when VRNI feature "Virtual Infrastructure latency" monitoring is enabled.
- Within NSX, when there are multiple VTEP per host - each VTEP is linked with a corresponding VMNIC. This is not the case for BFD traffic however. BFD Tunnel traffic from both VTEP's will leave the ESXi host based on the VXLAN routing table, using the source VMK interface specified for that destination network.
- Due to the BFD behavior, traffic is leaving multiple VMNICs with the same source IP address (VTEP IP), but different MAC addresses (MAC address of the source VMK)
- Dataplane learning on the physical network picks up on these packets and detects that VTEP IP address is seen from multiple mac addresses. Directs traffic to either interface based on the latest mac address detected.
Example #1 - Based on output below - we will see BFD traffic leaving from both VMK, with the same source VTEP IP addresses, which will cause the conflict.
[root@exmplesxi01:~] esxcli network ip route ipv4 list -N vxlan
Network Netmask Gateway Interface Source
-------- -------- --------- --------- ----------
default 0.0.0.0 10.0.0.254 vmk3 DHCP
10.1.1.0 255.255.255.0 0.0.0.0 vmk4 MANUAL
Example #2 - Based on output below - we will see BFD traffic all leaving from a single VMK. However, we will still experience the conflict, due to standard VXLAN traffic using the other VMNIC.
[root@exmplesxi02:~] esxcli network ip route ipv4 list -N vxlan
Network Netmask Gateway Interface Source
-------- -------- --------- --------- ----------
default 0.0.0.0 10.0.0.254 vmk4 MANUAL
10.1.1.0 255.255.255.0 0.0.0.0 vmk4 MANUAL