BFD tunnels down after a port scan against ESXi TEP interfaces

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

• ESXi host BFD tunnels have gone down either briefly or are permanently down
• ESXi hosts have multiple TEP interfaces on same subnet
• A network port scan has been run from another network against the ESXi TEP interfaces
• vmk11 is observed to have responded to the port scan using its own IP but from vmk10 interface and using vmk10 MAC address
• BFD traffic is still being sent correctly from vmk11 interface with vmk11 IP and MAC

Environment

VMware NSX-T Data Center

 

Cause

• When a TEP interface on an ESXi host is port scanned it will receive a SYN packet.
• The ESXi host may need to respond with a RST packet.
• When the source of that scan is on another network, the TEP interfaces response will need to be routed.
• Since TEP interfaces, e.g. vmk10 and vmk11, are on the same subnet, they have the same gateway.
• The TEP network stack routing table is currently consulted to determine how to reply
  
  #esxcli network ip route ipv4 list -N vxlan
Network        Netmask        Gateway          Interface  Source
-------------  -------------  ---------------  ---------  ------
default        0.0.0.0        ###.###.###.254  vmk10      MANUAL
###.###.###.0  255.255.255.0  0.0.0.0          vmk10      MANUAL
  
  In this case the default gateway is on vmk10.
• Therefore when vmk11 replies to a TCP port scan packet it will do so through the vmk10 interface.
• The result is a packet with vmk11 IP address and vmk10 MAC address sent on vmk10's interface.
• This behavior does not impact on datapath for encapsulated Geneve traffic.
• However if the physical network fabric uses these packets to update its ARP table instead of ARP snooping alone then it can poison the ARP table. This can then indirectly result in overlay datapath traffic disruption. 

Note: Since the behavior is ESXi related, NSX for vSphere environments can also experience network disruption after a port scan when multiple VTEPs in the same subnet are configured on ESXi hosts.

Resolution

This issue involving SYN packets is resolved in VMware ESXi 7.0 Update 3f

To prevent this issue from occurring, ensure that network port scans are not run against the ESXi host TEP interface subnets.