Error: Purple Screen Of Death (PSOD): DRS Advanced option "MaxMemMBHeadroomPerHost"

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article provides information on the DRS advanced option "MaxMemMBHeadroomPerHost" and the steps on how to apply it if required to do so.

Symptoms:

 

• Total reserved memory on the ESXi host shows as > 99.9% in a DRS enabled cluster
• The ESXi userworlds (like hostd, vpxa etc.) fail/crash due to Out Of Memory (OOM) errors.
• Purple Screen Of Death (PSOD) backtrace

VMware ESXi 8.0.3 [Releasebuild-24280767 x86_64]
NOT_IMPLEMENTED bora/vmkernel/sched/memsched_kmem.c:1436
PanicvPanicInt@vmkernel
Panic_NoSave@vmkernel
Panic_OnAssertAt@vmkernel
Int6_UD2Assert@vmkernel
gate_entry@vmkernel
MemSchedKmem_TransferMemory@vmkernel
VisorFSPagerUncompressPage@vmkernel
VisorFSMPNPagerGetPage@vmkernel
VisorFSGetPageRef@vmkernel
FSSVec_GetPageRef@vmkernel
UserMem_HandleMapFault@vmkernel
User_ArchExceptionHandleFault@vmkernel
User_Exception@vmkernel
Int14_PageFault@vmkernel
gate_entry@vmkernel

Environment

VMware vSphere ESXi 8.x

VMware vSphere ESXi 7.x

VMware vCenter Server 7.x

VMware vCenter Server 6.7.x

Cause

DRS uses the available memory capacity as advertised by the ESXi host to place workload VMs. This is smaller than the physical memory capacity of the host. If DRS places workload VMs and consumes all the memory reservation that is available on the host, any new process or existing process on the ESXi host may fail to allocate memory.

Resolution

Workaround:

DRS has an advanced option 'MaxMemMBHeadroomPerHost' which can be used to configure the headroom that DRS needs to set aside to handle dynamic changes in ESXi resource usage.
You can reserve up to a maximum of 5% of host memory and capped at the DRS advanced option. By default DRS reserves 128 MB as headroom reservation in MB for each ESXi host in the cluster. This headroom option is used in vCenter Server, the host doesn't have any knowledge of this headroom. So if a VM or any host system process requires memory, then the memory from headroom can still be utilized. 

Additionally growth in ESXi resource usage can eat into the headroom. DRS will do its best to again grow this headroom by updating how it divvies out resources and its VM placement. The resource pool status and the DRS faults indicate when DRS is unable to do so.
 

To set the DRS advanced option perform the following steps from the vSphere Webclient:
1. Right-click the DRS cluster and click Settings > vSphere DRS > Edit > Advanced Options > Add
2. In the Option column enter:
MaxMemMBHeadroomPerHost ​​
(Note: Do not change the case of MaxMemMBHeadroomPerHost)
3. Click in the Value column to enter the desired headroom value in MB, then click OK for this setting to take effect.