ESXi host network redundancy lost, vmnic reporting "packets completion seems stuck, issuing reset"

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi showing not responding in vCenter and having error message "Network uplink redundancy lost"
• vMotion stuck at 17% and eventually fails
• VMs unreachable over the network
• Intel x710 10GbE adapters used with i40en driver

/var/run/log/vobd.log shows vmnics flapping:

In(14) vobd[2097859]:  [netCorrelator] 28244825us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up
In(14) vobd[2097859]:  [netCorrelator] 28251037us: [vob.net.vmnic.linkstate.down] vmnic vmnic5 linkstate down
In(14) vobd[2097859]:  [netCorrelator] 28820376us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up

Following ESXI host logs shows packets completion stuck messages and hardware reset:

        /var/run/log/vmkwarning.log

Wa(180) vmkwarning: cpu16:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6714: vmnic5 : scheduler(0x431bb10c9f10)/device(0x4307bc0321c0) 0/1 lock up [stopped=0]:
Wa(180) vmkwarning: cpu16:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6722: detected at ##2234#### while last xmit at ##2234#### and 173081 bytes in flight [window 172920 bytes]
Wa(180) vmkwarning: cpu16:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6729: and last enqueued/dequeued at ##2234####/##2234#### [stress 0]
Wa(180) vmkwarning: cpu16:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6734: with 852 pkts inflight
Wa(180) vmkwarning: cpu34:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6769: vmnic5: packets completion seems stuck, issuing reset

/var/run/log/vmkernel.log

In(182) vmkernel: cpu103:2097645)i40en: i40en_ResetHW:15051:####:##:##.#: HW Resetting
In(182) vmkernel: cpu56:2098395)i40en: i40en_ResetHW:15051: ####:##:##.#: HW Resetting
In(182) vmkernel: cpu59:2097645)i40en: i40en_ResetHW:15051: ####:##:##.#: HW Resetting
In(182) vmkernel: cpu5:2098369)i40en:  i40en_ResetHW:15051: ####:##:##.#: HW Resetting

Environment

 VMware vSphere ESXi 8.0

Cause

Intel x710 firmware not responding and issuing reset messages.

Resolution

Engage Intel to investigate this further.

Additional Information

vmnicX down / up alerts in cluster with message "packets completion seems stuck, issuing reset" in logs

Broadcom | VMware | Hardware Compatibility Guide

Japanese KB: ESXi ホストでネットワークの冗長性が失われ、vmnic が "packets completion seems stuck, issuing reset" を報告する