ESXi host experiences PSOD due to datastore utilization at 100%

Products

• VMware vSphere ESXi

Issue/Introduction

• PSOD is triggered on an ESXi host when a datastore is at 100% utilization.
• The datastore however is in maintenance mode.

Reviewing the PSOD core dump, Unmap module is called when accessing the VMFS datastore.
 
VMware ESXi 8.0.3 [Releasebuild-24585383 x86_64]
#PF Exception 14 in world 2099311:hostd-worker IP 0x4200158441d2 addr 0xa8
Module(s) involved in panic: [vmfs3 Version VMFS6:24.82, VMFS6-L:24.82, VMFS5:14.81, VMFS5-L:14.60, Built on: MM DD YYYY]
cpu30:2099311 Backtrace for current CPU #30, worldID=2099311, fp=0x431513f01a00
cpu30:2099311 [0x4200158441d2]Unmap_RemoveFileFromProcessing@esx#nover+0x12 stack: 0x4539a3e1aad0, 0x43150ff99c00, 0x4539a3e1aac0, 0x43150fff1c40, 0x431513f01a00
cpu30:2099311 [0x42001579178c]PB3_ReinitCache@esx#nover+0x39 stack: 0x43150ffe8a10, 0x43150fff1c40, 0x430c92a23b70, 0x20210000, 0x43150ffe8a10
cpu30:2099311 [0x4200157e4e37]Fil3ExtendSparseIntVMFS6@esx#nover+0x8cc stack: 0x43150ff9a201, 0x3, 0x43150ff99c00, 0x43150ffaa8b0, 0x43151481a62c
cpu30:2099311 [0x4200157eb8ed]Fil3_SetFileLengthVMFS6@esx#nover+0xc66 stack: 0x100000001, 0x420015787cf4, 0x43150ffe8a10, 0x43150fff1c40, 0x40001010001
cpu30:2099311 [0x42001575aae8]Fil3_ExtendSysFile@esx#nover+0xf1 stack: 0x20000000, 0x431510002b28, 0x431510002800, 0x10000, 0x43150ffe8a10
cpu30:2099311 [0x4200158137f9]Res3ExtendResourcesIntVMFS6@esx#nover+0x3ea stack: 0x43150ff99c00, 0x4539a3e1adb8, 0x10110000, 0x9fff00, 0x43150fff1c40
cpu30:2099311 [0x4200158157d5]Res3ExtendResourcesVMFS6@esx#nover+0x4e6 stack: 0x4315153aa610, 0x300000000, 0x4539a3e1ae38, 0x43150ff99c00, 0x9fff00
cpu30:2099311 [0x42001581db24]Res3ExtendResourcesWrapperVMFS6WithRetry@esx#nover+0x13d stack: 0xa3e1ae20, 0x453900000000, 0x6432392d44494452, 0x100000000, 0xa0014000a0014
cpu30:2099311 [0x42001581deca]Res3_GrowResourceVMFS6@esx#nover+0x193 stack: 0xa01301, 0x10000000000, 0x0, 0x10000, 0x9fff00
cpu30:2099311 [0x42001575fcc1]Fil3_HandleOutOfResourcesRetry@esx#nover+0x2da stack: 0x0, 0x0, 0x0, 0x1, 0x430c92a407d0
cpu30:2099311 [0x4200157605d1]Fil3_CreateAndOpenFileWithRetry@esx#nover+0x42a stack: 0x4539a3e1bd40, 0x4539a3e1b286, 0x0, 0xd000000015, 0x0
cpu30:2099311 [0x4200142424d4]FSSVec_CreateAndOpenFile@vmkernel#nover+0x21 stack: 0x4539a3e1b270, 0x4200148bacc7, 0x4539a3e1b170, 0x4200148bac42, 0x430c8f842170
cpu30:2099311 [0x4200148bacc6]FSS_CreateAndOpenFile@vmkernel#nover+0x1bf stack: 0x430c8f842170, 0x4200148070ac, 0x77829b06, 0x430c92a407d0, 0x161fe8
cpu30:2099311 [0x4200147e51eb]UserFileOpen@vmkernel#nover+0x1c8 stack: 0x4539a3e1bd40, 0x45b9da039330, 0x4315000000c2, 0x4539a3e1b400, 0xffffffffffffffff
cpu30:2099311 [0x4200147ff1b8]UserObjTraversePath@vmkernel#nover+0x159 stack: 0xa, 0x432aac402010, 0x4539a3e1b514, 0x420000000002, 0x43150ffe8a10
cpu30:2099311 [0x4200147ff6c6]UserObj_TraversePathAt@vmkernel#nover+0x8b stack: 0x100, 0x0, 0x2e6b736964726268, 0x400000000, 0x39362d3934653831
cpu30:2099311 [0x4200148004df]UserObj_OpenAt@vmkernel#nover+0x54 stack: 0x0, 0x45b9da1bc840, 0x57, 0x432aac430dc0, 0x2e6b736964726268
cpu30:2099311 [0x420014800583]UserObj_FDOpenAt@vmkernel#nover+0x88 stack: 0x432aac402650, 0x2ac402010, 0x432aac402650, 0x0, 0x8180
cpu30:2099311 [0x4200147c4edd]LinuxFileDesc_OpenAt@vmkernel#nover+0x11e stack: 0x4900000000, 0x432aac476ae0, 0x432aac402010, 0x4539a3e1bf40, 0x0
cpu30:2099311 [0x4200147cfddc]User_LinuxSyscallHandler@vmkernel#nover+0xd9 stack: 0x3, 0x0, 0x0, 0x42001489b0c7, 0x103
cpu30:2099311 [0x42001489b0c6]gate_entry@vmkernel#nover+0xa7 stack: 0x0, 0x101, 0x48fc79a0a2, 0xc2, 0x8180 

Environment

VMware vSphere ESXi 8.x

Cause

The PSOD is caused due to a race condition in the Unmap_RemoveFileFromProcessing module when trying to create and open a file.

Resolution

Broadcom engineering is aware of this issue and currently there is no workaround/fix available.
This would be addressed in a future ESXi release.