Error: "Appliance is running low on memory. Add more memory to the machine" or Memory exhaustion on vCenter Server Appliance 7.x/8.x

Products

• VMware vCenter Server

Issue/Introduction

•  vCenter Server recently upgraded or installed to 7.x/8.x.
•  VAMI of vCenter Server Appliance will show Overall Health & Memory in Alert Status with the message: Appliance is running low on memory. Add more memory to the machine.

• High (more than 1GB) percpu allocator memory usage ("Percpu:" line in /proc/meminfo)
  # grep -i percpu /proc/meminfo

• High number (100+) of VMware Pod service restarts, verified using the below command on VCSA SSH session:

# journalctl -b 0 | grep "Started VMware Pod" | wc -l

Environment

VMware vCenter Server 7.x

VMware vCenter Server  8.x

Cause

High percpu memory usage is caused by huge number of dying memory cgroups as result of frequent "VMware Pod" service restart. The concern of dying memory cgroups was discussed on Linux Plumbers 2020. The percpu memory overhead is especially high on VM with CPU Hot Add enabled.

Resolution

This issue is resolved in vCenter Server 7.0 U2c. For more information, see VMware vCenter Server 7.0 Update 2c Release Notes

Workaround:
To workaround this issue :

1. Stop and disable vmware-pod service so the percpu memory usage won't grow further.
   • Login to VCSA using SSH
   • Execute the below commands to stop and disable "vmware-pod" service (Note: It is safe to disable vmware-pod service in on-prem environments)
     • systemctl stop vmware-pod
     • systemctl disable vmware-pod
       
2. [Immediate action] - no reboot required: drop caches to free dirty pages which will release hanging memory cgroups in dying state. Most of percpu memory will be freed.
   • Login to VCSA using SSH
   • Execute below command to drop caches
     • echo 3 > /proc/sys/vm/drop_caches
       
3. [Recommended] Disable kernel memory accounting for cgroups. Reboot required.
   • Login to VCSA using SSH
   • Edit file /boot/photon.cfg using VI editor:
     # vi /boot/photon.cfg
   • Add cgroup.memory=nokmem parameter to the end of  photon_cmdline=
     
   • Save the file using 'Esc + wq!' and exit VI editor
   • Reboot VCSA
4. [Optional] Disable CPU Hot Add of VCSA VM if not needed. VM power off required.
   • Shutdown the vCenter Server Appliance Virtual Machine
   • Right-click a virtual machine in the inventory and select Edit Settings.
   • On the Virtual Hardware tab, expand CPU, and uncheck Enable CPU Hot Add.
   • Click OK
   • Power ON the VC VM