ESXi vmnic link state flapping with the error - "Queue 0 of device vmnicX stuck, resetting the device"

Products

• VMware vSphere ESXi

Issue/Introduction

• In the vCenter, for the problematic host, we may see the alarm "Network Uplink Redundancy Lost" triggered.
  
  
• In the host's /var/log/vobd.log, we can see the below message reporting link state to be down:
  
  [netCorrelator] 15037473322098us: [vob.net.vmnic.linkstate.down] vmnic vmnicX linkstate down
  
  
• Around the same time, in the host's /var/log/vmkernel.log, we can see that, the queue of the vmnic got stuck and therefore, the watchdog issued a reset causing the link state to go down:
  
  cpu33:2097231)WARNING: Uplink: 21014: Queue 0 of device vmnicX stuck, resetting the device
cpu38:2097388)nmlx4_en: vmnic5: nmlx4_en_UplinkReset - (nmlx4_en_uplink.c:2577) Watchdog process reset the uplink for port: 0x1

Environment

VMware vSphere ESXi

Cause

As the queue of the vmnic adapter got stuck, the watchdog process reset the uplink. Generally, this is a recoverable state and the link state should be back up in a few seconds.

Resolution

• Obtain the network adapter driver name, version and firmware version using the steps detailed in Determining Network/Storage firmware and driver version in ESXi.
• From Supported drivers and firmware versions for I/O devices, compare the latest available driver and firmware version with the current version obtained from the above step. If the current version is outdated, kindly update the driver and firmware version to the latest supported.
• For driver upgrade, follow the steps from Download and install async drivers in VMware ESXi. For firmware upgrade, engage your hardware vendor.
• Even after upgrading the driver and firmware version to the latest, should the issue persist, engage your hardware vendor.