ESXi fails with PSOD (purple screen of death) "PANIC bora/vmkernel/main/dlmalloc.c:4928 - Usage error in dlmalloc"

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Host in a vSAN  cluster PSODs with the below backtrace

****-**-********.090Z cpu31:7709578 opID=ca308aea)World: 3355: PRDA 0x420047c00000 ss 0x0 ds 0x103 es 0x103 fs 0x103 gs 0x0
****-**-********.090Z cpu31:7709578 opID=ca308aea)World: 3357: TR 0x758 GDT 0x45394020f888 (0xffff) IDT 0x42000849d000 (0xffff)
****-**-********.090Z cpu31:7709578 opID=ca308aea)World: 3359: CR0 0x80010031 CR3 0xd55f603000 CR4 0x146768
****-**-********.126Z cpu31:7709578 opID=ca308aea)Backtrace for current CPU #31, worldID=7709578, fp=0x0
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09b520:[0x420007f7bbc0]PanicvPanicInt@vmkernel#nover+0x20c stack: 0xc3a73c687a63e3, 0x420007f7bbc0, 0x0, 0x420000000001, 0x420007f7bbc0
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09b5d0:[0x420007f7c33c]Panic_NoSave@vmkernel#nover+0x4d stack: 0x453a6d09b630, 0x453a6d09b5f0, 0x0, 0x4200084f337c, 0x1340
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09b630:[0x420007f32bd9]DLM_free@vmkernel#nover+0x642 stack: 0x4324baa10480, 0x420007f3b1ee, 0x4324baa10480, 0x4314a6420790, 0x453a6d09b9c0
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09b650:[0x420007f3b1ed]Heap_Free@vmkernel#nover+0xb2 stack: 0x453a6d09b9c0, 0x4314a6420790, 0x453a6d09b9c0, 0x0, 0x0
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09b6a0:[0x42000a1e71f8][email protected]#0.0.0.1+0xf9 stack: 0x1, 0x100000001, 0x6000000001, 0x2700000027,0x87
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09b8f0:[0x42000a1ea857][email protected]#0.0.0.1+0x424 stack: 0x0, 0x75b9f, 0x453a6d09ba42, 0x7e2e1bf2, 0x0
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09bad0:[0x420007e55141]DevFSIoctl@vmkernel#nover+0x372 stack: 0x4314a64207d0, 0x430d29fe9ea0, 0x180000012, 0x42000a1e6e5e, 0x430ae4624390
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09bb90:[0x420007e42677]FSSVec_Ioctl@vmkernel#nover+0x1c stack: 0x4314a6422e00, 0x0, 0x43130b601660, 0x1, 0x43130b601660
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09bba0:[0x4200084bb5ae]FSSObjectIoctlCommon@vmkernel#nover+0xff stack: 0x43130b601660, 0x1, 0x43130b601660, 0x0, 0x2
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09bc70:[0x4200084bb49f]FSS_IoctlByFH@vmkernel#nover+0xa4 stack: 0x35a19ff0a0, 0x4314a6420790, 0xbef, 0x0, 0x35a19ff0a0
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09bcc0:[0x4200083e609a]UserFileIoctl@vmkernel#nover+0x5fb stack: 0x38, 0x4314a6402010, 0x4314a6402010, 0x0, 0x4314a6402010
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09be40:[0x42000842109b]UserVmfs_Ioctl@vmkernel#nover+0x28 stack: 0x420047c00000, 0x4314a641e710, 0x453a6d09bf40, 0x420008435254,0xaa
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09bee0:[0x4200083c8211]LinuxFileDesc_Ioctl@vmkernel#nover+0x52 stack: 0x453a6d09bf40, 0x0, 0x10, 0x4200083cfddd, 0x0
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09bf10:[0x4200083cfddc]User_LinuxSyscallHandler@vmkernel#nover+0xd9 stack: 0x3, 0x0, 0x0, 0x42000849b0c7, 0x103
****-**-********.126Z cpu31:7709578 opID=ca308aea)0x453a6d09bf40:[0x42000849b0c6]gate_entry@vmkernel#nover+0xa7 stack: 0x0, 0x10, 0x359bfbb407, 0x35a19ff0a0, 0x0
****-**-********.144Z cpu31:7709578 opID=ca308aea)

Environment

• VMware vSAN 8.x 

Cause

• The error Corruption in dlmalloc suggests a memory corruption issue in the dynamic memory allocation subsystem (dlmalloc) which may lead to corrupting VMFS memory heap. This is a low-level memory allocator used by the ESXi kernel.

Resolution

Fixed in  ESXi 8.0 Update 3e.

VMware ESXi 8.0 Update 3e Release Notes