The host encountered PSOD and did not connect back after the reboot

Products

• VMware vSphere ESXi

Issue/Introduction

The ESXi host encountered a PSOD with errors about the VMNIC hardware drivers and did not connect back after rebooting.

PSOD Backtrace:

 cpu54:3455243)@BlueScreen: #PF Exception 14 in world 3455243:qfle3_sp_hel IP 0x42002310d4a3 addr 0x4521ddffffff  PTEs:0x800003a023;0x0;
 cpu54:3455243)Code start: 0x420022000000 VMK uptime: 330:14:42:21.431
 cpu54:3455243)0x45392089bc40:[0x42002310d4a3]qfle3_dcbx_read_mib@(qfle3)#<None>+0x3b stack: 0xf72838a01b7986
 cpu54:3455243)0x45392089bc50:[0x42002310d679]qfle3_dcbx_read_shmem_neg_results@(qfle3)#<None>+0x152 stack: 0x0
 cpu54:3455243)0x45392089bcf0:[0x42002310efca]qfle3_dcbx_pmf_update@(qfle3)#<None>+0xc3 stack: 0x0
 cpu54:3455243)0x45392089bd50:[0x4200230bad8b]qfle3_link_status_update@(qfle3)#<None>+0xc stack: 0x4520e79c8000
 cpu54:3455243)0x45392089bd70:[0x4200230e1354]qfle3_attn_int_deasserted@(qfle3)#<None>+0x1e09 stack: 0x431606d75870
 cpu54:3455243)0x45392089bea0:[0x4200230e2ff6]qfle3_sp_helper_func@(qfle3)#<None>+0x857 stack: 0x0
 cpu54:3455243)0x45392089bf40:[0x4200220da23f]HelperQueueFunc@vmkernel#nover+0x2d8 stack: 0x4539208a0b48
 cpu54:3455243)0x45392089bfe0:[0x4200223b3b79]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
 cpu54:3455243)0x45392089c000:[0x4200220c4c1f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
 cpu54:3455243)base fs=0x0 gs=0x42004d800000 Kgs=0x0

 

Cause

 

• /var/run/log/vmkernel.log may have the following warning for the active VMNICs:

 

 cpu59:2098264)WARNING: qfle3: qfle3_parity_attn:16971: [vmnic0] Parity errors detected in blocks:
 cpu59:2098264)qfle3: qfle3_parity_attn:16994: [vmnic0]
 cpu59:2098264)WARNING: qfle3: qfle3_parity_error:22292: [vmnic0] ESX_REC_PRT DS:0x1c RS:0 ES:0x4 to:5 rec:1 cr:0.
 cpu59:2098264)WARNING: qfle3: qfle3_trigger_grcdump:20646: [vmnic0] Enterd fwdmp_flags 0x31 ES 0x4
 cpu59:2098264)WARNING: qfle3: qfle3_trigger_grcdump:20709: [vmnic0] Triggered GRC dump cxt:1 st:0
 cpu59:2098264)WARNING: qfle3: qfle3_trigger_grcdump:20716: [vmnic0] GRC flags:0x931 trigger:1
 cpu59:2098264)qfle3: qfle3_attn_int_deasserted:18731: [vmnic0] Parity Error Occurred
 cpu59:2098264)WARNING: qfle3: qfle3_igu_int_disable:194: [vmnic0] Proper val not read from IGU!
 cpu50:2098226)qfle3: qfle3_recovery_driver_data_collect:20630: [vmnic0] Starting driver debug data dump.
 cpu50:2098226)qfle3: qfle3_get_drv_data_dump:20376: [vmnic0] Captured driver data dump size 0x3e6c4 hdr_size 0x40 dbg_data_sizez 0x3e680
 cpu50:2098226)qfle3: qfle3_recovery_driver_data_collect:20635: [vmnic0] Driver Debug Data Dump saved at 0x431607f785d0.
 cpu48:2098265)WARNING: qfle3: qfle3_acquire_hw_lock:688: [vmnic0] lock_status 0xffffffff  resource_bit 0x1

 

• This is an indication of the abrupt failure of the network adapter.
• Since the network is not detected, review the system hardware by enabling the ESXi shell by referring the following KB:  Using ESXi Shell in ESXi.
• Validate if the network adapter is present by executing the command: esxcfg-nics -l

[root@esxi:~] esxcfg-nics -l
Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description

• If the network cards are not detected, the output will be empty as above.
• To further verify, log in to the BMC of the host (iLO of HPE / iDRAC of DELL) and check for the network adapter status.

In the below screenshot, the status of the LOM is unknown, which is an indication of a hardware issue.

Example:

Resolution

Involve the hardware vendor for further investigation.