High disk usage observed on VMware AVI Load Balancer controller leader node

Products

• VMware Avi Load Balancer

Issue/Introduction

• High disk usage on the controller leader node.
• The disk usage goes over 80%.
• Run the following command to identify directories consuming the most disk space:
• SSH to the controller leader node. 
  • sudo du -Sh /host | sort -rh | head -n 10
    
    
• Review the output. In affected cases, the directory consuming the most disk space will appear as below:
  • /var/lib/postgresql/14/pg_metrics

Environment

• VMware AVI Load Balancer.
  • 30.1.1
    
  • 22.1.5

Cause

• The issue occurs due to a connection leak where the Metrics Manager leaves PostgreSQL database connections unclosed.

• Over time, this leads to excessive data accumulation in the metrics database, specifically in:

  • /var/lib/postgresql/14/pg_metrics

• This directory consumes significant disk space, causing overall disk usage on the controller to cross 80%.

Resolution

Fixed versions.

• 22.1.5-2p2
• 22.1.6
• 30.2.1
• 30.1.2
  

 

Workaround:

1. As a temporary workaround, restarting the process supervisor on the controller leader could resolve it for now, but it might occur in the future.
2. SSH to the controller leader (CLI).
3. sudo systemctl stop process-supervisor.service
4. sudo systemctl start process-supervisor.service

 

Note: The UI may take some time to show up when restarting the process supervisor on the controller.