vmnic down / up alerts in cluster with message "packets completion seems stuck, issuing reset" in logs

Products

• VMware vSphere ESXi

Issue/Introduction

• Frequent alerts regarding vmnic going down and up are reported in the cluster when using icen driver version 1.14.2.0
• vmkwarning.log shows lock and reset messages on the nic:

YYYY-MM-DDThh:mm:ss.014Z Wa(180) vmkwarning: cpu0:2375364)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7044: vmnic1 : scheduler(0x43230bccaf10)/device(0x430908414ec0) 0/1 lock up [stopped=0]:
YYYY-MM-DDThh:mm:ss.014Z Wa(180) vmkwarning: cpu0:2375364)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7052: detected at 1972366287 while last xmit at 1972360991 and 438510 bytes in flight [window 432432 bytes]
YYYY-MM-DDThh:mm:ss.014Z Wa(180) vmkwarning: cpu0:2375364)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7059: and last enqueued/dequeued at 1972366272/1972360990 [stress 0]
YYYY-MM-DDThh:mm:ss.014Z Wa(180) vmkwarning: cpu0:2375364)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7064: with 196 pkts inflight
YYYY-MM-DDThh:mm:ss.014Z Wa(180) vmkwarning: cpu0:2375364)WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7099: vmnic1: packets completion seems stuck, issuing reset

• vmkernel.log shows brief link down/up messages:

YYYY-MM-DDThh:mm:ss.014Z In(182) vmkernel: cpu106:2097709)netschedHClk: NetSchedHClkNotify:5067: vmnic1: link down notification
[....]
YYYY-MM-DDThh:mm:ss.015Z In(182) vmkernel: cpu106:2097709)NetqueueBal: 5211: vmnic1: device Up notification, reset logical space needed

Environment

VMware ESXi 8.0

Cause

 The driver is missing updates required for proper integration with the RDMA component

Resolution

Upgrade icen driver to 2.1.1.0 version.

 

 

Additional Information

Download and install async drivers in VMware ESXi