ESXi reporting memory Swap in and swap out rates despite overall low memory utilization

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts are reporting swap in and swap out rates above 0 constantly in the performance charts in vSphere Client, similar to the example below:
  
  
  
  
• The overall memory utilization of the ESXi host is low.
• External / third party monitoring tools are generation alarms for memory swap.
• Logs don't show any signs of other memory reclamation methods such as ballooning being used, despite the vmmemctl driver being enabled in VMware Tools.

Environment

• VMware vSphere ESXi 7.0.x
• VMware vSphere ESXi 8.0.x

Cause

Outside of overall memory shortages, ESXi will also swap memory pages to a swap file on a datastore when it cannot satisfy the amount of memory requested by a virtual machine due to specifically configured memory limit. If the guest operating system inside a virtual machine requests a higher amount of memory than is configured as memory limit for the virtual machine, anything beyond the limit will need to be swapped, resulting in an increase swap in\out rate.

Resolution

• To confirm the issue, start esxtop and open the memory tab using the "m" key
• The information after "MEM overcommit avg" shows if there is any memory oversubscription, values higher than 0 mean that more memory has been configured for virtual machines than is available as physical RAM
• The SWCUR column will display the amount of memory which is current swapped
• The SWTGT column shows the swap target, aka the amount of memory that needs to swapped overall
• The columns SWR/s and SWW/s show the rates with which memory is current read from or written to the swap file
• For example:

• Notice the "high state" at the end of the 3rd line. This state means that ESXi has ample memory available. ESXi knows different memory states, depending on how much free memory is available, High state > Clear state > Soft State > Hard State > Low State
• High state means that the host itself has ample memory free
• The ESXi does nonetheless swap
• This means that in the example the host would have enough free memory, but it cannot assign it to one or more virtual machines, indicating that these VM(s) are memory-limited.
• To confirm this, review the VM settings of all virtual machines running on the host. Expand "Memory" and check the Limit:

 

 

• Alternatively you can review the .vmx configuration file and look for the option "sched.mem.max"
• This setting defines the maximum amount of memory that the VM can use. Even if the VM is configured with a larger amount of memory, the limit restricts the VM to a maximum of this value.
• To stop the VM from swapping, change this limit to Unlimited and reload the VMX using Reloading a vmx file without removing the virtual machine from inventory

Note: Please be aware that even after applying these steps, ESXi will not immediately swap in every single memory page that has been swapped out earlier. Instead it will do this gradually over time. To confirm that the memory is being swapped back in, compare the SWCUR and SWTGT columns in esxtop and verify that SWTGT is smaller than SWCUR

Additional Information

For more troubleshooting, refer:

vSphere Monitoring and Performance