The link status of several vmnics transitions to a down state after a firmware upgrade of Broadcom NIC

Products

• VMware vSphere ESXi

Issue/Introduction

It is seen in /var/run/log/vmkernel.log that a link down notification is triggered due to a timeout when the bnxtnet NIC driver issues the HWRM_PORT_PHY_QCFG command after a firmware upgrade to 236.1.155.0

<DATE_TIME> Wa(###) vmkwarning: cpu#:#######)WARNING: bnxtnet: hwrm_send_msg:###: [vmnic# : ############] HWRM cmd resp_len timeout, cmd_type ##(HWRM_PORT_PHY_QCFG) seq ### <DATE_TIME> In(###) vmkernel: cpu#:#######)netschedHClk: NetSchedHClkNotify:####: vmnic#: link down notification

 

The driver and firmware versions are as follows:

Driver Info:          NICDriverInfo:             Bus Info: ####:##:##:#             Driver: bnxtnet             Firmware Version: 236.1.153.0 /pkg 236.1.155.0             Version: 236.1.128.0

Environment

VMware vSphere ESXi

Cause

We suspect that there might be an underlying bug or issue with this specific firmware version.

Resolution

1. As a first step, we recommend downgrading the firmware version.

2. Hardware vendor must be engaged for further troubleshooting/investigation related to network interface card failure.