ESXi host encounters a PSOD with PB3Manager@esx in backtrace

Products

• VMware vSphere ESXi

Issue/Introduction

• The host encounters a PF Exception 14 PSOD with the following backtrace:

MCSLockWork@vmkernel
PB3Manager@esx
CpuSched_StartWorld@vmkernel
Debug_IsInitialized@vmkernel

• An ESXi host may encounter a PSOD some time after "Out of memory" messages from VMFS, which could be triggered by fstrim command inside a VM and other memory-demanding file system workflows:

####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu##:4921524)WARNING: Heap: 3645: Heap vmfs3 already at its maximum size. Cannot expand.
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu##:4921524)WARNING: Heap: 3645: Heap vmfs3 already at its maximum size. Cannot expand.
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu##:4921524)WARNING: Heap: 4105: Heap_Align(vmfs3, 8192/8192 bytes, 512 align) failed.  caller: 0x##########
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu##:4921524)WARNING: Heap: 3645: Heap vmfs3 already at its maximum size. Cannot expand.
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu##:4921524)WARNING: Heap: 4105: Heap_Align(vmfs3, 8504/8504 bytes, 8 align) failed.  caller: 0x##########
####-##-##T##:##:##.###Z In(182) vmkernel: cpu##:4921524)Fil6: 3797: <Datastore Name>: <FD c49 r0> - Failed to unmap file blocks 20/32:Out of memory
####-##-##T##:##:##.###Z cpu41:2106945)Fil6: 4289: '<Datastore Name >': Fil6 file IO (<FD c40 r1>) : Out of memory

 

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

 

Cause

This PSOD is triggered by pointer block cache handling in the rare condition that the vmfs3 memory heap is exhausted.

Resolution

The VCF Broadcom Engineering Team is aware of this issue and a resolution is expected in a future release.