Alert: "continuous memory utilization" on vCenter VM causing continuous alarms and warnings

Products

• VMware vCenter Server

Issue/Introduction

• There are alerts about high memory utilization on the vCenter VM in the environment.
  
  
• When checking top, used memory looks normal, but there is a high amount of buffer/cache memory
  
  
  
  
• Checking slabtop indicates that radix_tree_mode has a sizable cache
  
  
  
  
• Checking the VAMI for performance charts of the vCenter, there is a trend over the last quarter that indicates an increase in memory usage:
  
  
• Checking the OS journalctl, the following event is present:
  
   vCenter Memory Resource status changed from Green to Yellow on [vcenter VM name} for continuous Memory utilization 85 percent in 10.0 mins
  
  

Cause

This is by design and there is no performance impact or effect on the environment.

The reason is that a process or processes in linux OS is performing a lot of file operations. This high amount of file operations at the OS level causes bloat of the buffer/cache memory. Linux will automatically flush the buffer/cache memory when pressure is put on the OS by applications. If no pressure is put on the operating system by applications for additional memory resources, then the buffer/cache will remain at a higher value until the memory is manually synced / flushed or the operating system is rebooted. 

Resolution

In order to further understand what is utilizing the memory, a script has been provided by engineering to get more memory statistics:

monitor_memory.py

The default values that the script monitors for are:

Memory threshold (--threshold): 80%
monitor Interval (--interval): 5 seconds
monitor duration (--duration): 1 hour

Once you have the output, open an SR with Broadcom Support for further investigation.

Workaround:

• Reboot the operating system

If the operating system cannot be rebooted, the following command can be run in the guest operating system:

sync; echo 3 > /proc/sys/vm/drop_caches 

Note: The sync command may have a performance impact on the operating system at the time it is run. It is recommended to run this during a maintenance window or off normal hours to minimize the impact.