Random Network Connectivity Loss on ESXi Hosts (vmnic Link Down) due to bnxtnet Firmware Crash

Products

• VMware vSphere ESXi

Issue/Introduction

 

• ESXi hosts experience a random loss of network connectivity.

• Specific network interfaces (e.g., vmnic1, vmnic3) report a physical link down status.

• Network adapters become unresponsive, causing the driver to mark the links as "down".

 

Environment

VMware vSphere ESXi

 

Cause

The network adapter firmware crashes or becomes unresponsive.
This triggers HWRM (Hardware Resource Manager) command timeouts within the bnxtnet driver.
When the driver fails to get a response from the firmware after multiple attempts, it registers a crash, dumps the firmware trace, and reports the link as down to the host OS.

Resolution

The combination of driver 236.1.128.0 and firmware 236.1.155.0 is considered valid per the Broadcom compatibility guide, updating the firmware resolves this specific crash.

1. Engage Hardware Vendor: Contact your hardware vendor (e.g., Dell) to validate and supply the updated firmware version.

2. Apply Firmware Update: Upgrade the NIC firmware to 236.1.173.0 (or newer, as recommended by your hardware vendor). This alternate version contains a specific fix for the "ESXi vmnic down following hot reset event" bug.
   
   
   

 

Additional Information

Broadcom NIC Firmware Crash resulting in Link Down (Error 0x89021)

Dell Reference : https://www.dell.com/support/home/en-uk/drivers/driversdetails?driverid=ff94t

Fixes:

- Allow 100G (4x25G), 25G (1x25G), and 200G (8x25G) links to be established without FEC enabled.
- Communication issue with Advanced NPAR, MX9116n, and SmartFabric.
- Connection errors on BCM5750x when vSAN RDMA enabled.
- Windows driver 233.0.148.0 cannot start with firmware 36.11.56.00.
- Packet loss with BCM57504 and SFP+ to BASE-T transceiver.
- ESXi vmnic down following hot reset event.