ESXi host configured as an NSX transport node experiences unexpected and intermittent outages, possibly at specific times of day.

Products

• VMware NSX

Issue/Introduction

• NSX is Federated with at least two Local Manager sites. 
• NSX prepared ESXi host experiences partial or complete outage on its uplink supporting its NSX TEP. 
  • This outage is not observed on uplinks not used by NSX. 
• Capturing traffic on the uplink used by the NSX TEP, example pktcap-uw command below, during the time of the outage and viewed in Wireshark shows the host receiving a packet sourced from north of NSX and destined for a VM at a different NSX Federation site. A large number of what Wireshark calls TCP Retransmissions are observed following each of these packets.  
  
  pktcap-uw --uplink vmnic# --capture UplinkRcvKernel,UplinkSndKernel -c 5000 --ng -o /path/to/filesystem/$(hostname)_$(date +"%FT%TZ")_UplinkKernels.pcapng
  
  Wireshark example:
  Inbound packet seen at UplinkRcvKernel. The Src and Dst IP's under the "Internet Protocol Version 4 header corresponds to the Edge TEP IP and the affected Host TEP IP respectively. 
  
  
  Outbound packet seen at UplinkSndKernel. Now the Internet Protocol Version 4 header shows the affected Host TEP IP as the Src and other hosts with TEPs in the same subnet for the Dst.
  
  
  
  • Looking closer at Wireshark, the first packet is observed entering the host at UplinkRcvKernel in the Packet comments, and each of the TCP Retransmissions are observed on UplinkSndKernel as the host is sending them out. 
  • Further, looking at the Geneve outer header it can be seen that each of the TCP Retransmissions are destined to the TEP IP of the other NSX prepared ESXi hosts on its same TEP subnet. 
• Viewing netstats (example command below) for the affected vmnic during the time of the issue may show a very large number of 'txeps' (Transmit Errors per Second). 
  
  net-stats -i 30 -ticqQWS -A > /path/to/filesystem/netstats
  
  Example:
  netstats:{"name": "vmnic#", "switch": "DvsPortset-#", "id": 224###6981, "mac": "f4:##:##:##:##:ee", "rxmode": 0, "tunemode": 2, "uplink": "true", "ens": "false", "promisc": "false", "sink": "false" ,
netstats  "txpps": 141851, "txmbps": 1694.7, "txsize": 1493, "txeps": 900278.62, "rxpps": 80838, "rxmbps": 905.5, "rxsize": 1400, "rxeps": 0.00,

Environment

VMware NSX 3.x

Cause

The traffic described above is due to designed behavior for BUM replication mode Hierarchical Two-Tier (MTEP). 

The cause of why the NSX Edge was sending this traffic south as BUM traffic in the first place rather than across the RTEP to the other Federated site where the destination VM lived is unknown at the time of this writing.  

Resolution

Workaround

• This behavior stopped occurring once the NSX Edges were upgraded to version 4.2.3.2.