High Memory usage on Linux based Virtual Machines

Products

• VMware vSphere ESXi

Issue/Introduction

• Linux based Virtual Machines report high Memory usage within the Guest OS.
• Running sar -r command on the Linux server reports approximately 100% memory usage.
   
  Example:
  
  
• The parameters are defined as below:
  
  
  • kbavail:
    An estimate of how much memory (in kilobytes) is available for starting new applications without swapping. This is usually a better indicator of available memory than kbmemfree because it takes into account page caches and reclaimable memory.
  • kbmemused:
    The amount of used memory + cached memory in kilobytes.
  • %memused:
    The amount of used memory + cached memory in percentage.
  • kbcached:
    The amount of memory (in kilobytes) used by the kernel to cache data (page cache). Reading from the cache is much faster than reading from a disk.
    
    
• However, correlating the data with the Virtual Machine Memory utilization using Aria Operations for ESXi host metrics reports moderate consumption.

Environment

VMware ESXi 8.x

VMware ESX 9.x

Cause

• The discrepancy between the in-guest memory reporting (99%) and vSphere metrics is due to the guest operating system aggressively using free memory for filesystem caching.

• For the above example, below is the calculation of memory utilization of the Virtual Machine at 4:50 am:

  • kbmemfree: 941,324 KB
  • kbmemused: 130,525,720 KB
  • kbbuffers: 97,520 KB
  • kbcached: 57,070,464 KB
    
    
  • Total Memory : 941,324 (kbmemfree) + 130,525,720 (kbmemused) = 131,467,044 KB
  • %memused: [130,525,720 (KBmemused) / 131,467,044 (total) ]* 100 = 99.28400...%
  • Actual memory usage: 130,525,720 (kbmemused) - 57,070,464 (kbcached) - 97,520 (kbbuffers) = 73,357,736 KB
    
    
• The Guest Operating System's %memused metric includes the cached memory, whereas vSphere's Active Memory tracks only pages actively being read or written.
• On the ESXi/ESX host, Virtual Machine's memory usage is observed using two different metrics:
  
  
  1. Consumed Memory (This includes cache):
     This is the amount of physical RAM on the ESXi host that has been allocated to the Virtual Machine.
     Once ESXi gives that memory page to the Virtual Machine, it considers it "Consumed." Even if the guest OS later frees up that memory or uses it just for cache, ESXi/ESX does not take it back (unless there is severe host memory contention and it is forced to use ballooning or swapping).
     
     
  2. Active Memory (This excludes idle cache):
     ESXi/ESX checks how many memory pages the Virtual Machine has actively read from or written to over the last few minutes.
     While file caches are "used" by the guest OS, they often sit idle for long periods. Because they aren't being actively read or written to every second, ESXi/ESX will report a very low Active Memory metric, even if Consumed memory is very high.

Resolution

• Linux operating systems are designed to utilize unused RAM for disk caching to improve read/write performance.
• This cached memory is instantly reclaimable by the kernel if an application requests it. Hence high %memused inclusive of cache does not indicate memory contention.
• To accurately determine if the Virtual Machine is starved for memory from the vSphere perspective, monitor the "Active Memory" counter, or check for non-zero values in "Memory Ballooning" or "Swapping" metrics, rather than relying solely on in-guest total used memory.