ESXi host network redundancy lost, vmnic reporting "packets completion seems stuck, issuing reset"

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi showing not responding in vCenter and having error message "Network uplink redundancy lost"
• vMotion stuck at 17% and eventually fails
• VMs unreachable over the network
• Intel x710 10GbE adapters used with i40en driver

 

vobd.log shows vmnics flapping:

In(14) vobd[2097859]:  [netCorrelator] 28244825us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up
In(14) vobd[2097859]:  [netCorrelator] 28251037us: [vob.net.vmnic.linkstate.down] vmnic vmnic5 linkstate down
In(14) vobd[2097859]:  [netCorrelator] 28820376us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up

vmkernel.log and vmkwarning.log shows packets completion stuck messages and hardware reset:

vmkwarning.log

Wa(180) vmkwarning: cpu16:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6714: vmnic5 : scheduler(0x431bb10c9f10)/device(0x4307bc0321c0) 0/1 lock up [stopped=0]:
Wa(180) vmkwarning: cpu16:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6722: detected at 8822346108 while last xmit at 8822341024 and 173081 bytes in flight [window 172920 bytes]
Wa(180) vmkwarning: cpu16:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6729: and last enqueued/dequeued at 8822342208/8822341025 [stress 0]
Wa(180) vmkwarning: cpu16:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6734: with 852 pkts inflight
Wa(180) vmkwarning: cpu34:2098871)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:6769: vmnic5: packets completion seems stuck, issuing reset

VMkernel.log

In(182) vmkernel: cpu103:2097645)i40en: i40en_ResetHW:15051: 0000:98:00.1: HW Resetting
In(182) vmkernel: cpu56:2098395)i40en: i40en_ResetHW:15051: 0000:98:00.0: HW Resetting
In(182) vmkernel: cpu59:2097645)i40en: i40en_ResetHW:15051: 0000:31:00.0: HW Resetting
In(182) vmkernel: cpu5:2098369)i40en: i40en_ResetHW:15051: 0000:31:00.1: HW Resetting

 

Environment

 VMware ESXi 8.0

Cause

Intel x710 firmware not responding and issuing reset messages.

Resolution

Engage Intel to investigate this further.

Additional Information

vmnicX down / up alerts in cluster with message "packets completion seems stuck, issuing reset" in logs