Memory balancer restarting Avi Controller processes due to high cache memory usage

Products

• VMware Avi Load Balancer

Issue/Introduction

Issue:

High Memory on Avi Controllers

 

Symptoms:

Constantly generated controller service failure events, such as below.

CONTROLLER_SERVICE_FAILURE

Description:

Controller service portal_resource_monitor_worker failed

cluster_service_failed_details:

• service_name: portal_resource_monitor_worker
• node_name: node2.controller.local

 

• These process failures are due to memory balancer restarting them as the host memory crossed the threshold of 85%.
• The memory balancer restarts the least priority process. This is done in order to bring down the memory consumption withiout affecting critical processes.
• The reason for high memory may be due to high cache utilization on controller nodes
  
  

How to verify that the issue is related to high cache memory:

• Run the command - free -w -h
• If the cache memory is more than 20% of total memory.

Environment

May occur in all cloud types

Cause

In 22.1.x, there is an issue with accounting related to memory management in the Controller for memory held in buffers and caches.

Resolution

• Bug ID - AV-181805
• Release Notes - Issue with accounting related to memory management in the Controller for memory held in buffers and caches.
• Workaround - Run the below command to free up cache memory
  
  

  free && sync && echo 3 > /proc/sys/vm/drop_caches && free

• Fix Version - 22.1.3-2p11, 22.1.4-2p1, 22.1.5, 30.1.1