Virtual machines experience random network disconnections on ESXi hosts using Broadcom BCM57454 or BCM57416 network adapters

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machines experience random intermittent network loss or dropped connections.

• The ESXi host utilizes Broadcom NetXtreme-E BCM57454 or BCM57416 network adapters using the bnxtnet driver.

• ESXi host network statistics display an abnormally high rate of Large Receive Offload (LRO) aborts, dropped receive packets, and physical link-down events on the active uplinks.

• Example output from the host network statistics visible in commands/nicinfo.sh.txt:

  rx_link_down_events: 9
        rx_stat_err: 279
      Receive packets dropped: 26220
      ...
        [rxq1] LRO aborts rx: 582150971
  
  
  rx_link_down_events: 8
        rx_stat_err: 311
      ...
        [rxq64] LRO aborts rx: 596192031
        [rxq65] LRO aborts rx: 603482894
        [rxq66] LRO aborts rx: 635856974

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi

Cause

This issue occurs when the ESXi host is running a bnxtnet driver version that is incompatible with the currently installed physical adapter firmware. Operating an unsupported driver and firmware combination causes the Broadcom network adapters to fail when processing Large Receive Offload (LRO) segments. This failure manifests as dropped receive packets and physical link flaps at the hardware level, which disrupts virtual machine network traffic on the virtual switch.

Resolution

This is a condition that may occur in a VMware vSphere ESXi environment.

 

To resolve this issue, upgrade the firmware on the Broadcom network adapters to the exact versions mandated by the VMware Compatibility Guide for your installed bnxtnet driver.

1. Identify the currently installed bnxtnet driver version on the affected ESXi host.

2. Consult the VMware Compatibility Guide to verify the exact required firmware version validated for that specific driver and ESXi build.

3. Upgrade the physical adapter firmware to match the validated configuration.

Additional Information

VMware Compatibility Guide (I/O Devices)