PSOD due to a VMFS datastore corruption

Products

• VMware vSphere ESXi

Issue/Introduction

Host purple screens with a Backtrace resembling the below trace:

[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)Backtrace for current CPU #28, worldID=2112420, fp=0x420023ff4600
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x4528c01c2cf0:[0x420023aff107]PanicvPanicInt@vmkernel#nover+0x327 stack: 0x4528c01c2dc8, 0x4302768052e8, 0x420023aff107, 0x420023ff4600, 0x4528c01c2cf0
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x4528c01c2dc0:[0x420023aff6b9]Panic_WithBacktrace@vmkernel#nover+0x56 stack: 0x4528c01c2e30, 0x4528c01c2de0, 0x4528c01c2e40, 0x4528c01c2df0, 0x1239265
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x4528c01c2e30:[0x420023afbe0c]NMI_Interrupt@vmkernel#nover+0x561 stack: 0x0, 0xf48, 0x0, 0x0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x4528c01c2f00:[0x420023b53392]IDTNMIWork@vmkernel#nover+0x7f stack: 0x420047000000, 0x420023b546dd, 0x0, 0x4528c01c2fd0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x4528c01c2f20:[0x420023b546dc]Int2_NMI@vmkernel#nover+0x19 stack: 0x0, 0x420023b4e068, 0x10b, 0x0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x4528c01c2f40:[0x420023b4e067]gate_entry@vmkernel#nover+0x68 stack: 0x0, 0x76006500440020, 0x43153a4315ac, 0x2a, 0x1a8
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9b7a0:[0x420024c39265]Fil3TruncateHelperVMFS6@esx#nover+0x1471 stack: 0x1a8000001a8, 0x0, 0x43153a43b800, 0x100, 0x100000
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9b8d0:[0x420024c3a733]Fil3TruncateVMFS6@esx#nover+0xe0 stack: 0x64, 0x27103a49ef80, 0x43153a49edf0, 0x420023b40e2d, 0x43153a49ee00
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9b940:[0x420024c3aaa3]Fil3DeallocateFileTxnVMFS6@esx#nover+0x110 stack: 0x43153a49ee00, 0x420000000000, 0x43153a65f43c, 0x43153a4c3200, 0x7e9ec67d541ee
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9b9b0:[0x420024c3b081]Fil3RemoveHelperVMFS6@esx#nover+0xda stack: 0x43153a7fde40, 0x1, 0x43153a4b1830, 0x1, 0x1
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9b9f0:[0x420024c418b0]Fil3RemoveVMFS6@esx#nover+0x1c09 stack: 0x43153a63c600, 0x43153a508001, 0x430aa93e6e01, 0x430aa93a3760, 0x43153a78c601
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9bd10:[0x420024bbd423]Fil3_Unlink@esx#nover+0x114 stack: 0x430aa93e6ef8, 0x1, 0x0, 0x0, 0x420023eb4b80
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9bd50:[0x420023a3ba6f]FSSVec_Unlink@vmkernel#nover+0x20 stack: 0x453934c9bdc0, 0x1, 0x0, 0x430aa93e6e90, 0x420023eb4b80
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9bd60:[0x420023a37541]FSS_Unlink@vmkernel#nover+0xaa stack: 0x0, 0x430aa93e6e90, 0x420023eb4b80, 0x453934c9bdc0, 0x453934c9bdc0
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9bda0:[0x420023f05c8d]UserFileUnlink@vmkernel#nover+0x26 stack: 0x420047005610, 0x45b945b6c918, 0x5345543030315355, 0x30302d5054465354, 0x7365732d32303030
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9bdb0:[0x420023ebba7e]UserObj_Unlink@vmkernel#nover+0x63 stack: 0x5345543030315355, 0x30302d5054465354, 0x7365732d32303030, 0x6d762e6573726170, 0x5345543000006b64
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9bee0:[0x420023f0850b]LinuxFileDesc_Unlink@vmkernel#nover+0x3c stack: 0x57, 0x453934c9bf40, 0x57, 0x420023eb4d24, 0xffffffffffffffef
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9bf10:[0x420023eb4d23]User_LinuxSyscallHandler@vmkernel#nover+0x1a4 stack: 0x0, 0x0, 0x0, 0x420023b4e068, 0x10b
[YYYY-MM-DDTHH:MM:SS] cpu28:2112420)0x453934c9bf40:[0x420023b4e067]gate_entry@vmkernel#nover+0x68 stack: 0x0, 0x57, 0xc8a96c7087, 0x0, 0xc86793f380

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

The PSOD is due to CPU hang, while trying to free blocks related to a corrupt VMFS datastore.

Resolution

The issue is due to a corrupt VMFS datastore. To identify the cause for corruption, initiate a case with Broadcom support.