ESXi with NIC driver qfle3/qedentv fails with a PSOD (purple screen of death)

Products

• VMware vSphere ESXi 8.0
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 6.0

Issue/Introduction

•  VMware ESXi host fails with a PSOD (purple screen of death)
•  ESXi physical adapters (vmnic) use driver qfle3/qedentv
•  MTU of 9000 is configured on the vmnic
•  The crash screen contains a backtrace similar to the below example:

Heap_AlignWithTimeoutAndRA
Pkt_AllocHandleWithSize
Pkt_AllocWithFlags
vmk_PktAllocForDMAEngine

 

Environment

VMware vSphere ESXi

Cause

When a pNIC driver attempts to allocate 9000 MTU size packets to replenish its receive queue, as system memory is fragmented, networking subsystem is not able to allocate large page memory to grow the memory pool that backs such packets. As a result, memory pool API spins for large pages for a long time, triggers CPU heartbeat check failure and crashes the host.

Resolution

This is a known issue impacting ESXi hosts.

Workaround: 

• Reserve memory pool for such pNIC drivers through command:

esxcli system settings kernel set -s netPktJumboSlabMinSize -v <SIZE_OF_MEMORY_TO_RESERVE_IN_MB>