High Disk Storage on one of the follower nodes

Products

• VMware Avi Load Balancer

Issue/Introduction

High Disk Storage on one of the follower nodes

Environment

Linux Server Cloud

Cause

1. SSH to the follower node

2. Login to controller bash of the follower node

admin@10-x-x-x:~$ sudo docker exec -it avicontroller bash

3. Look for directories/files consuming the most space. The below command display the same sorted by size

root@10-x-x-x:~# sudo du 2> >(grep -v '^du: cannot \(access\|read\)' >&2) -h / | sort -rh | head -n 40
347G    /
142G    /hostroot
121G    /hostroot/opt/avi
121G    /hostroot/opt
118G    /vol
118G    /hostroot/opt/avi/controller/data
118G    /hostroot/opt/avi/controller
80G     /vol/root2/var
80G     /vol/root2
80G     /var
80G     /hostroot/opt/avi/controller/data/root2/var
80G     /hostroot/opt/avi/controller/data/root2
78G     /vol/root2/var/lib
78G     /var/lib
78G     /hostroot/opt/avi/controller/data/root2/var/lib
68G     /vol/root2/var/lib/avi
68G     /var/lib/avi
68G     /hostroot/opt/avi/controller/data/root2/var/lib/avi
39G     /vol/root2/var/lib/avi/tech_support
39G     /var/lib/avi/tech_support
39G     /hostroot/opt/avi/controller/data/root2/var/lib/avi/tech_support
21G     /vol/root1/var
21G     /vol/root1
21G     /hostroot/opt/avi/controller/data/root1/var
21G     /hostroot/opt/avi/controller/data/root1
20G     /vol/root2/var/lib/avi/logs
20G     /var/lib/avi/logs
20G     /hostroot/opt/avi/controller/data/root2/var/lib/avi/logs
19G     /vol/root1/var/lib
19G     /hostroot/var/lib
19G     /hostroot/var
19G     /hostroot/opt/avi/controller/data/root1/var/lib
18G     /vol/root2/var/lib/avi/logs/ALL-EVENTS
18G     /var/lib/avi/logs/ALL-EVENTS
18G     /hostroot/var/lib/docker/overlay2
18G     /hostroot/var/lib/docker
18G     /hostroot/opt/avi/controller/data/root2/var/lib/avi/logs/ALL-EVENTS
17G     /vol/root1/var/lib/avi
17G     /vol/pkgs
17G     /hostroot/opt/avi/controller/data/root1/var/lib/avi
root@10-x-x-x:~#

As we can see in the output highlighted above, the /var/lib/avi/tech_support directory that stores the generated techsupport files and the /vol/pkgs directory that stores the software/upgrade pkgs are consuming 39 GB and 17 GB respectively. These files consume space on the disk and can be deleted to free up space.

Resolution

Clear out the techsupport files from /var/lib/avi/tech_support and old Avi images from under /vol/pkgs directory.