Intermittent packet drops for VMs on NSX overlay networks (VXLAN or Geneve) using Broadcom BCM5741x NetXtreme-E Network Adapters during high traffic load

Products

• VMware Cloud Foundation
• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

This article describes a known issue found in Broadcom network adapters running 214.0.xxx.x firmware versions.

Symptoms:

• The network adapters are Broadcom BCM57412, BCM57414 or BCM57416 with firmware version 214.0.166.0, 214.0.188.0, 214.0.190.0, 214.0.191.0 or other 214.0.xxx.x firmware
• Network traffic on NSX-V or NSX-T logical switches (VXLAN or Geneve) is dropped intermittently
• Packet drops can occur during periods of high network load
• In NSX-T environments, the BFD channel used to monitor the health of Geneve Tunnels between transport nodes may go down as reported in the Transport Nodes section of the NSX UI or as verified from the following command:
  nsxdp-cli bfd sessions list
  
  Remote              Local               local_disc       remote_disc          recvd                sent                local_state         local_diag          client                     flaps
  
  192.168.1.151     192.168.1.152        5c7ff9cb            0                   251                 296                 down                Control Detection Time Expiredvdl2                2        
  192.168.1.153     192.168.1.152        19ad9cd9            0                   237                 298                 down                Control Detection Time Expiredvdl2                2        
  192.168.1.154     192.168.1.152        7c11ce76            0                   248                 297                 down                Control Detection Time Expiredvdl2                2        
  
   

Environment

VMware vSphere ESXi 6.5
VMware NSX-T Data Center
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7

Cause

Packet drops occur on NSX overlay networks (VXLAN or Geneve) due to incorrect filter setup logic in the NIC firmware.

Resolution

To resolve this issue upgrade the NIC firmware:

• For Dell branded adapters, upgrade to firmware version 214.0.222.1
• For other hardware vendors, please refer to the vendor to obtain the appropriate release

Please refer to your hardware vendor for instructions to obtain and update the NIC firmware.