Intermittent packet drops for VMs on NSX overlay networks (VXLAN or Geneve) using Broadcom BCM5741x NetXtreme-E Network Adapters during high traffic load
search cancel

Intermittent packet drops for VMs on NSX overlay networks (VXLAN or Geneve) using Broadcom BCM5741x NetXtreme-E Network Adapters during high traffic load

book

Article ID: 343762

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware NSX Networking VMware vSphere ESXi

Issue/Introduction

This article describes a known issue found in Broadcom network adapters running 214.0.xxx.x firmware versions.

Symptoms:
  • The network adapters are Broadcom BCM57412, BCM57414 or BCM57416 with firmware version 214.0.166.0, 214.0.188.0, 214.0.190.0, 214.0.191.0 or other 214.0.xxx.x firmware
  • Network traffic on NSX-V or NSX-T logical switches (VXLAN or Geneve) is dropped intermittently
  • Packet drops can occur during periods of high network load
  • In NSX-T environments, the BFD channel used to monitor the health of Geneve Tunnels between transport nodes may go down as reported in the Transport Nodes section of the NSX UI or as verified from the following command:
    nsxdp-cli bfd sessions list

    Remote              Local               local_disc       remote_disc          recvd                sent                local_state         local_diag          client                     flaps

    192.168.1.151     192.168.1.152        5c7ff9cb            0                   251                 296                 down                Control Detection Time Expiredvdl2                2        
    192.168.1.153     192.168.1.152        19ad9cd9            0                   237                 298                 down                Control Detection Time Expiredvdl2                2        
    192.168.1.154     192.168.1.152        7c11ce76            0                   248                 297                 down                Control Detection Time Expiredvdl2                2        


     


Environment

VMware vSphere ESXi 6.5
VMware NSX-T Data Center
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7

Cause

Packet drops occur on NSX overlay networks (VXLAN or Geneve) due to incorrect filter setup logic in the NIC firmware.

Resolution

To resolve this issue upgrade the NIC firmware:
  • For Dell branded adapters, upgrade to firmware version 214.0.222.1
  • For other hardware vendors, please refer to the vendor to obtain the appropriate release
Please refer to your hardware vendor for instructions to obtain and update the NIC firmware.