Intermittent packet drops for VMs on NSX overlay networks (VXLAN or Geneve) using Broadcom BCM5741x NetXtreme-E Network Adapters during high traffic load

Products

• VMware Cloud Foundation
• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

• Network traffic on NSX-V or NSX-T logical switches (VXLAN or Geneve) is dropped intermittently
• Packet drops can occur during periods of high network load
• In NSX-T environments, the BFD channel used to monitor the health of Geneve Tunnels between transport nodes may go down as reported in the Transport Nodes section of the NSX UI or as verified from the following command:
  nsxdp-cli bfd sessions list
  
  Remote              Local               local_disc       remote_disc          recvd                sent                local_state         local_diag          client                     flaps
  
  192.168.###.###     192.168.###.###        #######                   251                 296                 down                Control Detection Time Expiredvdl2                2        
  192.168.###.###     192.168.###.###        #######            0                   237                 298                 down                Control Detection Time Expiredvdl2                2        
  192.168.###.###    192.168.###.###         #######            0                   248                 297                 down                Control Detection Time Expiredvdl2                2        
  
   

Environment

VMware NSX-T Data Center

VMware vSphere ESXi 6.7

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

The network adapters are Broadcom BCM57412, BCM57414 or BCM57416 with firmware version 214.0.166.0, 214.0.188.0, 214.0.190.0, 214.0.191.0 or other 214.0.xxx.x firmware

Cause

Packet drops occur on NSX overlay networks (VXLAN or Geneve) due to incorrect filter setup logic in the NIC firmware.

Resolution

To resolve this issue upgrade the NIC firmware:

• For Dell branded adapters, upgrade to firmware version 214.0.222.1
• For other hardware vendors, please refer to the vendor to obtain the appropriate release

Please refer to your hardware vendor for instructions to obtain and update the NIC firmware.

To confirm latest driver and firmware versions, see Broad Compatibility Guide I/O devices