PSOD (purple screen of death) due to multiple firmware command timeouts

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

6.7 or 7.0 ESXi has PSOD (purple screen of death) with the below backtrace 

Panic Details: Crash at 2021-02-25T13:03:11.939Z on CPU 33 running world 2098358 - vmnic7-pollWorldvmnic7_np[1]-0x4300d1cbd5c0. VMK Uptime:1:04:37:15.860
Panic Message: @BlueScreen: #GP Exception 13 in world 2098358:vmnic7-pollW @ 0x420027d3d1b4
Backtrace:
 0x45391d81bcd0:[0x420027d3d1b4]FastSlab_AllocWithTimeout@vmkernel#nover+0x40 stack: 0x420048400140, 0x45391d81bd48, 0x43125bf41080, 0x45391d81bde8, 0x45391d81bde0
 0x45391d81bd20:[0x420027c29631]vmk_PageSlabAlloc@vmkernel#nover+0x22 stack: 0x453900001000, 0x5d3e0a4000, 0x453900001000, 0xffffffffffffffff, 0x43125bf41080
 0x45391d81bd30:[0x420027dce4ed]PktPageAlloc_AllocPages@vmkernel#nover+0x3a stack: 0x453900001000, 0xffffffffffffffff, 0x43125bf41080, 0x1, 0x45391d81bda0
 0x45391d81bd80:[0x420027e73763]vmk_PktAllocPage@vmkernel#nover+0x10 stack: 0x45d95495e2c0, 0x409235e600, 0x7, 0x43125be0c080, 0x43125be01580
 0x45391d81bd90:[0x42002893a0a1]qfle3_page_alloc_and_map@(qfle3)#<None>+0x22 stack: 0x7, 0x43125be0c080, 0x43125be01580, 0x8, 0x43125be0c080
 0x45391d81bde0:[0x42002894ef6a]qfle3_alloc_rx_sge_mbuf@(qfle3)#<None>+0x2f stack: 0x43125be0c080, 0x11c, 0x43125be0c080, 0x11c, 0x1
 0x45391d81be20:[0x4200289507d5]qfle3_rxeof@(qfle3)#<None>+0x1012 stack: 0x103b007c006080a3, 0x100000001, 0x45d9ab749d00, 0x8, 0x43125be01580
 0x45391d81bf20:[0x42002895623b]qfle3_netpoll_rxcb@(qfle3)#<None>+0x1fc stack: 0x1, 0x0, 0x1, 0x0, 0x0
 0x45391d81bf70:[0x420027e7524d]NetPollWorldCallback@vmkernel#nover+0x17e stack: 0xd0000002b, 0x420028956040, 0x420027e75240, 0x0, 0x0
 0x45391d81bfe0:[0x420027f69481]CpuSched_StartWorld@vmkernel#nover+0x82 stack: 0x0, 0x420027cbe6a0, 0x0, 0x0, 0x0
 0x45391d81c000:[0x420027cbe69f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0

Environment

VMware ESXi 6.7.x
VMware vSphere ESXi 7.0.x

Cause

The issue depends upon some events happening in vmkernel. The main issue is that some unrelated modules in the vmkernel consume the CPU for longer than the expected duration which causes the qfle3 driver to fail with an unexpected path.

Resolution

This is resolved in QLogic qfle3 VMware ESX Native Driver for ESXi 6.5/6.7/7.0:

ESXi 6.5/6.7 Version Driver version 1.1.14.0 or above
ESXi 7.0 Version Driver version 1.4.14.0 or above