vmnic flaps due to TX hang in ESXI hosts causing network disruption

Products

VMware vSphere ESX 8.x

Issue/Introduction

ESXi hosts experience intermittent network connectivity drops or complete link down events on physical network adapters (vmnics)
The affected Network adapter (vmnic) is down or fails with a failed criteria code 128 the vobd log.

2026-07-01T17:14:01.080Z In(14) vobd[2098007]: [netCorrelator] 5844515590265us: [vob.net.dvport.uplink.transition.down] Uplink: vmnicX is down. Affected dvPort: vdrPort/XX XX XX XX XX XX XX XX-XX XX XX XX XX XX XX XX. 1 uplinks up. Failed criteria: 128

When reviewing the /var/run/log/vmkernel.log, you will see warnings from the network scheduler watchdog (netschedHClk) detecting that packet completion is stuck. This is immediately followed by a forced device reset and a link down notification.

vmkwarning: cpu94:2097613)WARNING: xxxxxxx: xxxxxxx_uplink_reset:8641: [vmnic0 : 0x43132de232d0] TX timeout! <<==========
vmkwarning: cpu89:18042712)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7099: vmnicX: packets completion seems stuck, issuing reset <<============
vmkernel: cpu94:2097613)xxxxxxx: xxxxxxx_uplink_quiesce_dev:2555: [vmnicX : 0x43132de232d0] Quiesce dev 0x43132de232d0
vmkernel: cpu94:2097613)netschedHClk: NetSchedHClkNotify:5067: vmnicX: link down notification

From the stats, you may see the affected vmnic experienced tx hang

NIC Private statistics:
      port tx_pause_frames: 0
      port rx_pause_frames: 12710
      tx timeout count: 3. <====

Environment

VMware vSphere ESXI 8.x

Cause

The ESXi network scheduler watchdog monitors the transmit (Tx) packet flow to the physical Network Interface Card (NIC). The logs indicate a Tx hang, where packets dispatched to the physical NIC are queuing up and not being processed or acknowledged by the hardware.

Because the NIC stops responding to transmit requests, the ESXi host's watchdog mechanism automatically issues a reset to the Broadcom adapter to attempt to recover the data path. This reset forces the driver to quiesce (pause) the device, resulting in a temporary or persistent "link down" state. This behavior is typically caused by a known firmware and driver bug or an incompatibility between the installed NIC driver and the physical adapter's firmware.

Resolution

To resolve this issue and prevent the watchdog from resetting the adapter, you must update the Broadcom NIC firmware and the ESXi NIC driver to a certified and compatible combination.

Identify your specific Broadcom network adapter model and your current ESXi version. Determining Network/Storage firmware and driver version in ESXi
Navigate to the VMware hardware compatibility guide https://compatibilityguide.broadcom.com
Download and apply the latest certified firmware and driver combination for your specific hardware.

Additional Information

ESXi host network redundancy lost, vmnic reporting "packets completion seems stuck, issuing reset

TX hang can happen due to spurious hardware errors, if this issue is happening in a cluster with multiple similarly configured host, please have the OEM run some hardware diagnostics with a focus on the NIC.