Cisco Servers configured with 'Cisco VIC Ethernet NIC' report to be down with the error 'WQ[0] error_status 4'

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vmnic link state goes down causing Network Outage:
  YYYY-MM-DDTHH:MM:SS: [netCorrelator] 2580862798036us: [vob.net.vmnic.linkstate.down] vmnic vmnic2 linkstate down

• We see below errors in vmkernel.log:
  YYYY-MM-DDTHH:MM:SS cpu6:95573)WARNING: nenic: enic_log_q_error:139: [0000:48:00.0] WQ[0] error_status 4
YYYY-MM-DDTHH:MM:SS cpu6:95573)WARNING: nenic: enic_isr_msix_err:190: [0000:48:00.0] Scheduled soft reset to recover from error
YYYY-MM-DDTHH:MM:SS cpu35:66532)nenic: enic_soft_reset_helper:3043: [0000:48:00.0] Resetting
YYYY-MM-DDTHH:MM:SS cpu35:66532)nenic: enic_soft_reset_helper:3049: [0000:48:00.0] Reset completed
YYYY-MM-DDTHH:MM:SS cpu35:66532)nenic: enic_link_check:261: [0000:48:00.0] Link DOWN
YYYY-MM-DDTHH:MM:SS cpu6:95573)WARNING: nenic: enic_log_q_error:139: [0000:48:00.0] WQ[0] error_status 4
YYYY-MM-DDTHH:MM:SS cpu6:95573)WARNING: nenic: enic_isr_msix_err:190: [0000:48:00.0] Scheduled soft reset to recover from error
YYYY-MM-DDTHH:MM:SS cpu35:66532)netschedHClk: NetSchedHClkNotify:2908: vmnic2: link down notification
YYYY-MM-DDTHH:MM:SS cpu35:66532)nenic: enic_soft_reset_helper:3043: [0000:48:00.0] Resetting
YYYY-MM-DDTHH:MM:SS cpu35:66532)nenic: enic_soft_reset_helper:3049: [0000:48:00.0] Reset completed

Environment

• VMware vSphere ESXi 8.x
• VMware vSphere ESXi 7.x

Cause

• This problem occurs due to a fault in the VIC (Cisco UCS Virtual Interface Card) hardware.

Resolution

• Involve Cisco TAC (Technical Assistance Center) to run Hardware Diagnostics for the NIC.