Intermittent vSAN Unicast Connectivity Failures and TX Timeouts on Broadcom bnxtnet Drivers

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• In the vSphere Client, go to the cluster Monitor tab > vSAN > Skyline Health. Select the vSAN: Basic (unicast) connectivity check error and click Troubleshoot. This will provide a report showing which specific hosts are unreachable and which VMkernel adapters are being used.
• vmkping tests across the vSAN VMkernel interface show intermittent timeouts (See Testing VMkernel network connectivity with the vmkping command).
• vSAN objects may remain healthy initially, but the cluster shows inconsistent connectivity between multiple hosts despite being in the same partition.
• The VMkernel Observation Daemon (/var/log/vobd.log) detected that the physical link state for the network adapter card (vmnic) dropped:

YYY-MM-DDTHH:MM:SS In(14) vobd[2098531]:  [netCorrelator] 104301077275us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic1 is down. Affected dvPort: 4/50 20 4c 28 d3 8a 05 91-22 6c eb f0 2f 43 14 bc. 1 uplinks up. Failed criteria: 128

Environment

VMware vSAN 8.x

VMware vSAN 9.x

Cause

The failure is triggered by a Transmit (TX) timeout condition within the Broadcom bnxtnet driver. A version mismatch between the async driver and the NIC firmware leads to transmit queue lockups, causing the ESXi network scheduler to issue a watchdog reset.

 

• In the /var/log/vmkernel.log file of the ESXi host  vmnic1 driver reports TX timeouts and subsequent uplink reset errors:

WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7101: vmnic1: packets completion seems stuck, issuing reset

WARNING: bnxtnet: bnxtnet_uplink_reset:9089: [vmnic1 : 0x45230f09e000] TX timeout!bnxtnet: bnxtnet_uplink_quiesce_dev:2606: [vmnic1 : 0x45230f09e000] Quiesce dev 0x45230f09e000Uplink: 65: update uplink: vmnic1, state: 0

 

Resolution

Resolution:

1. Identify Current Versions: Execute the following command via SSH or the ESXi Shell to confirm the current state, driver, and firmware versions of the affected network adapter: esxcli network nic get -n vmnicX
2. Verify Compatibility: Reference the VMware Compatibility Guide (VCG) Broadcom | VMware | Hardware Compatibility Guide to ensure your current driver and firmware versions are a validated match for your specific hardware.
3. Upgrade Firmware / Driver:Update the NIC firmware and/or driver to a supported, validated pairing per the VCG to resolve the underlying transmit (TX) timeout defect.

References:

Determining Network/Storage firmware and driver version in ESXi

Download and install async drivers in VMware ESXi

Additional Information

vmnic flaps and Tx Timeout observed on Broadcom bnxtnet driver

ESXi host network redundancy lost, vmnic reporting "packets completion seems stuck, issuing reset"