Cisco Servers configured with 'Cisco VIC Ethernet NIC' report to be down with the error 'WQ[0] error_status 4'

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vmnic link state goes down causing Network Outage:

2021-07-01T03:23:12.434Z: [netCorrelator] 2580862798036us: [vob.net.vmnic.linkstate.down] vmnic vmnic2 linkstate down

• We see below errors in vmkernel.log:

2021-07-01T03:23:12.425Z cpu6:95573)WARNING: nenic: enic_log_q_error:139: [0000:48:00.0] WQ[0] error_status 4

2021-07-01T03:23:12.425Z cpu6:95573)WARNING: nenic: enic_isr_msix_err:190: [0000:48:00.0] Scheduled soft reset to recover from error

2021-07-01T03:23:12.425Z cpu35:66532)nenic: enic_soft_reset_helper:3043: [0000:48:00.0] Resetting

2021-07-01T03:23:12.433Z cpu35:66532)nenic: enic_soft_reset_helper:3049: [0000:48:00.0] Reset completed

2021-07-01T03:23:12.433Z cpu35:66532)nenic: enic_link_check:261: [0000:48:00.0] Link DOWN

2021-07-01T03:23:12.433Z cpu6:95573)WARNING: nenic: enic_log_q_error:139: [0000:48:00.0] WQ[0] error_status 4

2021-07-01T03:23:12.433Z cpu6:95573)WARNING: nenic: enic_isr_msix_err:190: [0000:48:00.0] Scheduled soft reset to recover from error

2021-07-01T03:23:12.433Z cpu35:66532)netschedHClk: NetSchedHClkNotify:2908: vmnic2: link down notification

2021-07-01T03:23:12.434Z cpu35:66532)nenic: enic_soft_reset_helper:3043: [0000:48:00.0] Resetting

2021-07-01T03:23:12.440Z cpu35:66532)nenic: enic_soft_reset_helper:3049: [0000:48:00.0] Reset completed

Environment

VMware vSphere ESXi 6.5

Cause

• It is caused by VIC Hardware failure.

Resolution

• Involve Cisco TAC to run Hardware Diagnostics for the NIC.