Purple Screen of Death (PSOD) on ESXi with NIC driver qfle3/qedentv - VMware vSphere ESXi

Products

• VMware vSphere ESXi 8.0
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 6.0

Issue/Introduction

•  VMware ESXi host fails with a PSOD (purple screen of death)
•  ESXi physical adapters (vmnic) use driver qfle3/qedentv
•  MTU of 9000 is configured on the vmnic
•  The crash screen contains a backtrace similar to the below example:

Heap_AlignWithTimeoutAndRA
Pkt_AllocHandleWithSize
Pkt_AllocWithFlags
vmk_PktAllocForDMAEngine

Backtrace:
PCPU 64: SSSSSSSSSSSVVSSSSVSSUSSSSSSSSSSS
Code start: 8x42000f600000 VMK uptinc: YYYY-MM-DDT HH:MM:SS
Saved backtrace from: pcpu 94 Heartbeat NMI
0x453a6191bc88:[0x42000f78c929]Tiner_GetCycles@vmkernel#nover+0x2 stack: @x5a0457546a7a66
0x453a6191bc90:[0x42000f785681]SP_WaitLockIRQ@vnkernellnover+Oxce stack: 0x4200578082b8
0x453a6191bce0:[0x42000f7857b1]SPLockIRQWork@vnkernel#nover+0x5e stack: @x26ba
0x453a6191bd08:[0x42000f73b52f ]Heap_AlignWithTincoutAndRA@vmkernel#nover+0xf4 stack: 0x20b02530
0x453a6191bdB0:[0x42000f8d19c0]Pkt_AllocHandleWithSize@vmkernel#nover+0x1f1 stack: 0xd1ba
Bx453a6191bda8:[0x42000f8dlc51]Pkt_AllocWithFlags@vnkerncl#novcr*xc stack: Bxdlba
0x453a6191bdc0:[0x42000f965711]vmk_PktAllocForDMAEngine@vnkernel#nover+0x3a stack: 0x43181ff6cf80
0x453a6191bdc8:[0x4200189cfb65]qedentv_alloc_rx_buffer@(qedentv)#+0x52 stack: 0x2
0x453a6191be20:[0x4200109d274b]qedentv_rx_int@(qedentv)l+8x634 stack: 0x453a6191bec8
0x453a6191beb0:[0x4200109d3444]qedenty_poll0(qedentv)#+0x211 stack: 0x420041c00000
Jx453a6191bf28:[0x42000f89ddc0]NetPoll_PollPkts@vmkernel#nover+Bx49 stack: 0x42000f89ddb4
0x453a6191bf70:[0x42000f89eb77]NetPollWorIdCallback@vmkernel#nover+0xc4 stack: 0x453a68a1f000
0x453a6191bfc8:[0x42000fcdd93c]CpuSched_StartWor Id@vnkernel#nover+0xbf stack: 0x0
0x453a6191c000:[Ox42000f7453ef ]Debug_IsInitialized@vnkernel#nover+0xc stack: 0x0
basc fs=0x0 gs=0x420057800000 Kgs=0x8
3 other PCPUs are in panic.

Environment

VMware vSphere ESXi

Cause

When a pNIC driver attempts to allocate 9000 MTU size packets to replenish its receive queue, as system memory is fragmented, networking subsystem is not able to allocate large page memory to grow the memory pool that backs such packets.

As a result, memory pool API spins for large pages for a long time, triggers CPU heartbeat check failure and crashes the host.

Resolution

This is a known issue impacting ESXi hosts.

Workaround: Reserve the memory pool for the pNIC drivers by executing the following command on the affected ESXi host:

• Reserve memory pool for such pNIC drivers through command:

esxcli system settings kernel set -s netPktJumboSlabMinSize -v <SIZE_OF_MEMORY_TO_RESERVE_IN_MB>
Example- esxcli system settings kernel set -s netPktJumboSlabMinSize -v 1024 

Note: This is a kernel-level change and requires a host reboot to take effect.

After the host reboot, verify the setting with:
esxcli system settings kernel list -o netPktJumboSlabMinSize