Intermittent vSAN Unicast Connectivity Failures and TX Timeouts on Broadcom bnxtnet Drivers

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• In the vSphere Client, go to the cluster Monitor tab > vSAN > Skyline Health. Select the vSAN: Basic (unicast) connectivity check error and click Troubleshoot. This will provide a report showing which specific hosts are unreachable and which VMkernel adapters are being used.
• vmkping tests across the vSAN VMkernel interface show intermittent timeouts (See Testing VMkernel network connectivity with the vmkping command).
• vSAN objects may remain healthy initially, but the cluster shows inconsistent connectivity between multiple hosts despite being in the same partition.
• The VMkernel Observation Daemon (/var/log/vobd.log) detected that the physical link state for the network adapter card (vmnic) dropped:

YYY-MM-DDTHH:MM:SS In(14) vobd[2098531]:  [netCorrelator] 104301077275us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic1 is down. Affected dvPort: 4/50 20 4c 28 d3 8a 05 91-22 6c eb f0 2f 43 14 bc. 1 uplinks up. Failed criteria: 128

Environment

VMware vSAN 8.x

VMware vSAN 9.x

Cause

The failure is triggered by a Transmit (TX) timeout condition within the Broadcom bnxtnet driver. A version mismatch between the async driver and the NIC firmware leads to transmit queue lockups, causing the ESXi network scheduler to issue a watchdog reset.

 

• In the /var/log/vmkernel.log file of the ESXi host  vmnic1 driver reports TX timeouts and subsequent uplink reset errors:

WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7101: vmnic1: packets completion seems stuck, issuing reset

WARNING: bnxtnet: bnxtnet_uplink_reset:9089: [vmnic1 : 0x45230f09e000] TX timeout!bnxtnet: bnxtnet_uplink_quiesce_dev:2606: [vmnic1 : 0x45230f09e000] Quiesce dev 0x45230f09e000Uplink: 65: update uplink: vmnic1, state: 0

 

Resolution

Resolution:

1. Identify Current Versions: Execute the following command via SSH or the ESXi Shell to confirm the current state, driver, and firmware versions of the affected network adapter: esxcli network nic get -n vmnicX
2. Verify Compatibility: Reference the VMware Compatibility Guide (VCG) Broadcom | VMware | Hardware Compatibility Guide to ensure your current driver and firmware versions are a validated match for your specific hardware.
3. Upgrade Firmware / Driver:Update the NIC firmware and/or driver to a supported, validated pairing per the VCG to resolve the underlying transmit (TX) timeout defect.

References:

Determining Network/Storage firmware and driver version in ESXi

Download and install async drivers in VMware ESXi

Additional Information

ESXi host network redundancy lost, vmnic reporting "packets completion seems stuck, issuing reset"