Error: VMs are crashing with the message: "The virtual machine is allocating too many pages while system is critically low in free memory. It is forcefully terminated to prevent system instability.“

Products

• VMware vSphere ESXi

Issue/Introduction

On the ESXi hosts you can see messages similar to:

"Error message from #######.#####.#####.###: VMware ESX unrecoverable error: (vcpu-5) vmk: vcpu-0:VMM unresponsive to lowmem swap actions. :mpn alloc=3132 free=1347, :swap action post=0, tgt=12325756, reclaimed=6904556, :attempt swap=1024 balloon=0 cow=0 :done swap=182".

You also have bottlenecks on the ESXi hosts caused by virus scanner that performs full weekly scan of the VM guest OS

Environment

VMware vSphere 8

Cause

CPU overcommitment in the cluster.

If virus scanning is started for all VMs simultaneously, they will start inducing memory pressure by touching memory so memory levels on certain hosts might become low.

Given that CPUs are oversubscribed and all VMs become active doing virus scanning activity, vCPUs might be de-scheduled for extended periods of time to yield to other active vCPU threads.

This could potentially lead to slower response to memory reclamation requests.

Resolution

Workaround:

• Stage virus scanning within each host

• disable VMM NUMA migration, the context in which the issue is exposed, by adding:

  monitor_control.disable_numamigrate = "TRUE"

  to the *.vmx file of the VMs you want to protect, or to /etc/vmware/config of each host in the cluster to apply it to every VM.