LOG_FREQUENT_DISK_CLEANUP events are frequently noticed on AVI controller

Products

• VMware Avi Load Balancer

Issue/Introduction

• On the Avi Load Balancer Controller, a log disk usage event is generated every 10 minutes when disk utilisation exceeds predefined limits. The automated cleanup script subsequently removes certain log files, resulting in a LOG_FREQUENT_DISK_CLEANUP event being recorded in the controller interface.

Event details:

Description : Log disk usage exceeds limit ######## on controller #####.controller.local from #########. After 2 times of cleanup, the disk size is #######
 

disk_cleanup_event_details: 

controller: ######.controller.local
size_limit: ########
curr_size: ########
cleanup_count: 2
from_time: #############

Environment

Avi Load Balancer Version: 30.2.x

Cause

• In Older versions(22.x) even though log disk usage was above the threshold events were not generated. Event "LOG_FREQUENT_DISK_CLEANUP" is introduced from 30.2.x version only.
• If the logs disk usage (/var/lib/avi/logs/) on Avi Load Balancer Controller reaches above 70%(this calculation is done by controller using internal logic), cleanup script automatically deletes few files and an event is triggered in controller UI.

 

Resolution

Steps to Verify:

Login to Controller SSH (Bash) using admin credentials:

Run "sudo -i" command and enter the admin password as depicted below.

admin@########:~$ sudo -i
[sudo] password for admin: 
root@########:~#

• Execute the below command. It displays top top 30 folders with disk usage. 

• Looks for folders consuming more disk space under directory "/var/lib/avi/logs" 

• Command: sudo du 2> >(grep -v '^du: cannot \(access\|read\)' >&2) -h / | sort -rh | head -n 30

Note:

• Command output provide top 30 folders consuming more space. In the below output only the required folder information is added.
• To get rid of the event we have to remove files from folders within /var/lib/avi/logs/ (Don't delete any folders forcefully)

root@###########：~# sudo du 2> >(grep -v '^du: cannot \(access\|read\)' >&2) -h / | sort -rh | head -n 30
7.0G		/var/lib/avi/logs
4.7G		/var/lib/avi/logs/virtualservice-xxxxx-xxxxx-xxxx-xxxx-xxxxxxxx
2.5G		/var/lib/avi/logs/virtualservice-xxxxx-xxxxx-xxxx-xxxx-xxxxxxxx/se-xxxxxxxxxxx
2.2G		/var/lib/avi/logs/virtualservice-xxxxx-xxxxx-xxxx-xxxx-xxxxxxxx/se-xxxxxxxxxxx
5.2G        /var/lib/avi/log/ALL-EVENTS/se-########

From the output there are 4 folders consuming more disk space under directory "/var/lib/avi/logs"

Steps to Cleanup:

Navigate to each shortlisted folder from the above steps and check for files older than 30 days by executing below command.

Step-1: Verify the total number of files located within the folder.

Command: ls -ltrh | wc -l

root@#########:/var/lib/avi/logs/virtualservice-xxxxx-xxxxx-xxxx-xxxx-xxxxxxxx# ls -ltrh | wc -l

Step-2: Check for the files older than 30 days in the same folder 

This Command lists all the files older than 30 days

Command: find . -type f -mtime +30 -exec ls -ltrh {} +

root@#########:/var/lib/avi/logs/virtualservice-xxxxx-xxxxx-xxxx-xxxx-xxxxxxxx# find . -type f -mtime +30 -exec ls -ltrh {} +

This Command will provide count on files older than 30 days

Command: find . -type f -mtime +30 -exec ls -ltrh {} + | wc -l

root@#########:/var/lib/avi/logs/virtualservice-xxxxx-xxxxx-xxxx-xxxx-xxxxxxxx# find . -type f -mtime +30 -exec ls -ltrh {} + | wc -l

Step-3:  If the difference is more between Number of files in Step-1 and Step-2 then run the command to remove files older than 30 days.

If the difference between Number of files in Step-1 and Step-2 is less (Ex: between 10 or 100 files) then try to change the value to higher than 30 days.

Command: find . -type f -mtime +30 -exec rm -f {} +

root@#########:/var/lib/avi/logs/virtualservice-xxxxx-xxxxx-xxxx-xxxx-xxxxxxxx# find . -type f -mtime +30 -exec rm -f {} +

Note: If customer wants to keep data less than or more than 30 days data change the value accordingly. 


Conclusion:

We are enhancing log retention and cleanup efficiency in an upcoming Log Manager release 2.0 to better handle these scenarios.

Please reach out to support team before removing these files.