ESXi hosts might fail with purple diagnostic screen due to rare memory fragmentation issue

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Under rare conditions with high host memory usage, severe memory fragmentation can lead to network device driver memory allocation requests to take too long which triggers a heartbeat miss PSOD. The PSOD backtrace contains entries similar to:

2021-11-12T00:12:38.785Z cpu2:267947)Panic: 547: Panic from another CPU (cpu 2, world 267947): ip=0x420013cf7434 randomOff=0x13c00000:
2021-11-12T00:12:38.826Z cpu0:400001)Saved backtrace from: pcpu 2 Heartbeat NMI
2021-11-12T00:12:38.847Z cpu0:400001)0x45388951b990:[0x420013d42528]FastSlabAllocPages@vmkernel#nover+0xf9 stack: 0x420013d424f0
2021-11-12T00:12:38.853Z cpu0:400001)0x45388951bc20:[0x420013d429a8]FastSlabCreateNewBuffer@vmkernel#nover+0xe9 stack: 0x45388951bce0
2021-11-12T00:12:38.858Z cpu0:400001)0x45388951bc80:[0x420013d43600]FastSlabCreateObj@vmkernel#nover+0x4cd stack: 0x100000000
2021-11-12T00:12:38.864Z cpu0:400001)0x45388951bd30:[0x420013d43e2a]FastSlabReplenishCPU@vmkernel#nover+0x22f stack: 0x430074602ae8
2021-11-12T00:12:38.869Z cpu0:400001)0x45388951bd80:[0x420013d4510f]FastSlabAllocSlow@vmkernel#nover+0x84 stack: 0x9
2021-11-12T00:12:38.873Z cpu0:400001)0x45388951bda0:[0x420013dded5e]Pkt_SlabAllocPkt@vmkernel#nover+0x133 stack: 0x1f
2021-11-12T00:12:38.879Z cpu0:400001)0x45388951bde0:[0x420013dde281]Pkt_AllocHandleWithSize@vmkernel#nover+0xee stack: 0x4520c01a8850
2021-11-12T00:12:38.885Z cpu0:400001)0x45388951be00:[0x420013dde570]Pkt_AllocWithFlags@vmkernel#nover+0xd stack: 0x45388951bf20
2021-11-12T00:12:38.890Z cpu0:400001)0x45388951be20:[0x420013e82f71]vmk_PktAllocForDMAEngine@vmkernel#nover+0x46 stack: 0x0

Environment

VMware vSphere ESXi 7.0.0

Resolution

This issue is resolved with VMware ESXi 7.0 Update 3f (build number 20036589)