High Swap Memory Usage on Avi Controller Nodes

Products

• VMware Avi Load Balancer

Issue/Introduction

• Monitoring tools (such as SolarWinds or Zabbix or NagiOS or ManageEngine etc) report that the Controller leader node is utilizing nearly 100% of its swap memory. This high swap usage, despite available physical RAM (memory), may raise concern for customers.

Environment

• External monitoring tools reporting high swap usage for Avi Controller Nodes.

Cause

• The monitoring tools reports show that swap memory is nearly or fully consumed, while overall physical memory usage remains at a moderate level
• Example output from Avi Controller Node
  
  
  

  // stats_collection logs from controller nodes
  
  root@avi-controller:~# cat /var/lib/avi/log/stats_collection.log
  INFO [stat_collector.pprint_ntuple:216] host swap memory:  Total : 12.0G, Used : 12.0G, Free : 3.4M, Percent : 100.0%...
  INFO [stat_collector.pprint_ntuple:216] host memory:  Total : 31.4G, Available : 7.3G, Percent : 76.6%, Used : 17.8G...

• The high swap memory utilization is normal and expected behavior for Linux-based systems, including the Avi Controller, especially when configured with low swappiness.

  • Linux Memory Management: The Linux kernel moves memory pages that are very seldom used or considered "unlikely to be needed soon" to the swap space, even if ample physical RAM is available. This is a mechanism to free up physical RAM for more active data and disk buffers/cache, optimizing overall system performance.

  • Avi Swappiness Setting: Avi Controllers are configured with a low swappiness value of 1 (cat /proc/sys/vm/swappiness returns 1). This is the minimum possible value, meaning the kernel is very reluctant to swap, but it will still proactively move inactive memory pages to swap.

  • Non-Concerning Usage: Since the overall controller physical memory usage is only around 72%–76% (well below critical thresholds ~85%), the system is simply utilizing swap space efficiently to manage inactive data pages, which poses no impact to the customer or system performance.

  Swap space usage only becomes a concern when:

  1. RAM is fully consumed (approaching 85% or more consistently).

  2. The kernel is forced to continuously swap pages between RAM and disk ("thrashing"), which would result in high disk I/O activity.

Resolution

• The primary resolution is to monitor the overall physical memory usage instead of the swap memory usage.
• Do not be alarmed by 100% swap usage if the overall physical memory usage remains below 85%.
• If the overall physical memory usage consistently hovers around 85% or above, then it becomes a concern, and further investigation is required. At this point, advanced memory details can be collected using the command along with controller debug logs

  root@avi-controller:~#smem -tka
  root@avi-controller:~#smem -w -p
  root@avi-controller:~#smem -t -p
  root@avi-controller:~#smem -u -p
  root@avi-controller:~#free && sync && echo 3 > /proc/sys/vm/drop_caches && free
  root@avi-controller:~#top -b -o +%MEM | head -n 30
  root@avi-controller:~#ps aux --sort=-%mem | head -n 30
  

  
  
  

Workaround:

• Rebooting the Controller is a temporary workaround that will clear the swap memory, but the swap usage will gradually build back up as the Linux kernel continues its memory management process.

Additional Information