ESXi host becomes unmanageable due to insufficient available reservable memory

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
On ESXi 6.5 server containing virtual machines with large amount of memory reservation, in some cases, host might run out of reservable memory leading to::

• Various services crashing with core dumps
• Virtual machine becoming unresponsive
• Host becoming unmanageable from vCenter Server or otherwise
• Host becoming inaccessible via SSH (cannot login)
• Unable to collect log bundles (cannot spawn new processes) until one of the virtual machines is powered off.
• You will see various log files containing Out of memory events.

Environment

VMware vSphere ESXi 6.5

Cause

This issue occurs when host runs out of available reservation and unable to meet short term/run time demands by some of its services.

Resolution

This is a known issue affecting ESXi 6.5.x.

Currently, there is no resolution.

Workaround:
An ESXi host can run out of reservation if virtual machines consume all reservable memory, so we recommend leaving sufficient amount of reservable memory available so that other system groups (i.e. services) can get requested amount of memory during run time.

While creating virtual machines it is prudent to leave few GBs of available memory as non-reserved.

For changing reservation, see:

At vCenter Server level:

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vm_admin.doc/GUID-49D7217C-DB6C-41A6-86B3-7AFEB8BF575F.html

At VCD level:

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vm_admin.doc/GUID-9902FD4C-4727-43F7-AEAF-901DA4CB4D60.html