High Packet Retransmissions and Network Instability Due to Driver and Firmware Mismatch on BCM57414 Adapters - VMware vSphere ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Network performance monitoring tools (e.g., Dynatrace, etc) report high packet retransmission rates (typically >10%) on Windows or Linux virtual machines. These alerts correlate with network-side instability on Broadcom BCM57414 network adapters running on VMware vSphere ESXi.

Environment

• VMware vSphere ESXi 8.0 or later.
• Broadcom BCM57414 NetXtreme-E Network Adapter.

Cause

Network retransmissions are frequently caused by a misalignment between the bnxtnet driver and the physical adapter firmware. When the driver-to-firmware HWRM (Hardware Resource Manager) interface versions are out of sync, the NIC may experience buffer management inefficiencies or firmware-level hangs, leading to packet drops that present as retransmissions in guest OS monitoring tools.

Resolution

1. Identify the current driver and firmware versions by running the following command on the ESXi host:
   
   esxcli network nic list

esxcli network nic get -n vmnic#
Example: esxcli network nic get -n vmnic0

   
   
2. Obtain the VID, DID, SVID, and SSID.
   
   vmkchdev -l |grep vmnic#
Exmaple: vmkchdev -l |grep vmnic0
   
   
3. Consult the Broadcom Compatibility Guide to identify the certified driver and firmware stack for the specific BCM57414 hardware revision.
4. Contact server vendor support (Cisco, Dell, HPE, etc) to verify your findings.
5. If they approve do the following: 
   1. Place the ESXi host in Maintenance Mode.
      
      1. Do one of the following: 
         Update the Broadcom network adapter firmware to the certified version provided by the server OEM (e.g., ####).
         Install the certified bnxtnet driver version using the following command: esxcli software vib install -d /path/to/driver.zip
         Or
         Update ESXi host using vLCM Image which includes ESXi, driver, and firmware.
   2. Reboot the ESXi host to apply the firmware and driver changes.
   3. Verify the alignment of the new driver and firmware versions by re-running:
      
      esxcli network nic get -n vmnic#
      Example: esxcli network nic get -n vmnic0
      
      
   4. Exit Maintenance Mode and monitor network telemetry via network monitoring tool or vCenter performance charts to confirm the reduction in packet retransmissions.