Host hangs abruptly when using Cisco UCS VIC

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi host hangs abruptly when using Cisco UCS Virtual Interface Card (VIC).
• Host fails to respond to ping requests.
• VMs not reachable over the network
• DCUI login works, but does not respond to commands.
• Unable to see any hardware faults in the UCS Manager.

Environment

Because this issue is rooted in a hardware and driver communication fault with the Cisco UCS VIC (rather than a specific ESXi kernel bug), it affects the major current releases:

• VMware vSphere ESXi 7.x (All builds)
• VMware vSphere ESXi 8.x (All builds)

Cause

In the /var/run/log/vmkernel.log file, there will be error messages similar to the below lines.

The Cisco nenic driver loses communication with the vNIC hardware, and the queues get stuck.

WARNING: nenic: _vnic_dev_cmd2:331: 0000:62:00.0:Timed out devcmd 4

WARNING: nenic: _vnic_dev_cmd2:331: 0000:62:00.1:Timed out devcmd 4

WARNING: Uplink: 21913: Queue 0 of device vmnic2 stuck, resetting the device

ESXi attempts to recover the stuck vNIC by sending a reset command. The hardware fails to respond entirely, triggering a "surprise removal" (the OS interprets this as the hardware being physically yanked out).

nenic: enic_uplink_reset:3440: [0000:62:00.3] Resetting

WARNING: nenic: _vnic_dev_cmd2:294: 0000:62:00.3: Fatal error while issuing devcmd2 command 36, hardware surprise removal

Because the reset failed, the driver forces the device offline, causing complete network isolation.

nenic: enic_quiesce_dev:920: [0000:62:00.3] disable device

WARNING: nenic: vnic_wq_disable:239: Failed to disable WQ[0]

Resolution

This is a known Cisco firmware bug tracked under Cisco Bug ID: CSCwp64942. VMware's official resolution is to engage your Cisco hardware vendor for diagnostics and to apply the appropriate firmware/driver updates.