BFD tunnels down after a port scan against ESXi TEP interfaces
Article ID: 324202
Updated On:
Products
Issue/Introduction
- NSX-T Data Center
- ESXi hosts have multiple TEP interfaces on same subnet.
- ESXi host BFD tunnels have gone down either briefly or are permanently down
- A network port scan has been run from another network against the ESXi TEP interfaces
- vmk11 is observed to have responded to the port scan using its own IP but from vmk10 interface and using vmk10 MAC address
- BFD traffic is still being sent correctly from vmk11 interface with vmk11 IP and MAC
Environment
Cause
When a TEP interface on an ESXi host is port scanned it will receive a SYN packet.
The ESXi host may need to respond with a RST packet.
When the source of that scan is on another network, the TEP interfaces response will need to be routed.
Since TEP interfaces, e.g. vmk10 and vmk11, are on the same subnet, they have the same gateway.
The TEP network stack routing table is currently consulted to determine how to reply
#esxcli network ip route ipv4 list -N vxlan
Network Netmask Gateway Interface Source
------------- ------------- --------------- --------- ------
default 0.0.0.0 192.168.141.254 vmk10 MANUAL
192.168.141.0 255.255.255.0 0.0.0.0 vmk10 MANUAL
In this case the default gateway is on vmk10.
Therefore when vmk11 replies to a TCP port scan packet it will do so through the vmk10 interface.
The result is a packet with vmk11 IP address and vmk10 MAC address sent on vmk10's interface.
This behaviour does not impact on datapath for encapsulated Geneve traffic.
However if the physical network fabric uses these packets to update its ARP table instead of ARP snooping alone then it can poison the ARP table. This can then indirectly result in overlay datapath traffic disruption.
Note: Since the behaviour is ESXi related, NSX for vSphere environments can also experience network disruption after a port scan when multiple VTEPs are configured on ESXi hosts.
Resolution
This issue involving SYN packets is resolved in VMware ESXi 7.0 Update 3f available at VMware Downloads.
Note VMware is aware of another problem scenario if a security scanner sends an unsolicited ACK to a TEP. This can trigger the same BFD tunnel down response. This is known issue currently under investigation.
Workaround:
To prevent this issue from occurirng ensure that network port scans are not run against the ESXi host TEP interface subnets.
Additional Information
Feedback
