High disk consumption on VIO controller

Products

• VMware Integrated OpenStack

Issue/Introduction

• VIO controller shows high disk consumption 
• Running 'df -h'on one of the controller shows the /dev/sda4 on '/ ' as 82% consumed.

 

 

Environment

• VMware Integrated Openstack 7.x

Cause

• This issue occurs if the logs in "other_vhosts_access.log"  inside the var/log/httpd directory gets accumulated and consumes a large amount of space 

[2025-05-07 14:12:29.106] /diff/var/log/httpd ]# ls -ltrh

[2025-05-07 14:12:31.535] total 22G
[2025-05-07 14:12:31.535] -rw-r--r-- 1 root root  1.9K  Feb 15  2023 error.log
[2025-05-07 14:12:31.535] -rw-r--r-- 1 root root   22G  May  7  05:12 other_vhosts_access.log

Resolution

• Determine which container the large size log file is located  and identify which container instance using the overlay folder name.
• Restart the pod so that the log file will be cleared once the pod is recreated. 

 

To find out which Pod the following file is in:

 

1. Check for disk space usage on the VIO controllers 

#df -h

  Note : Use the below command to find the large file.

          # find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10 

2. Navigate to directory where the largest log file exists. In this case we found "other_vhosts_access.log" was consuming 22GB of space.

#root@controller-xxxx [ /var/lib/docker/overlay2/e7fe54eb13417c94567708492e62a3aa7fc5dc311cd300e3ee42ae7dd6d5a621/merged/var/log/httpd ]# ls -ltrh
total 22G

-rw-r--r-- 1 root root 1.9K Feb 15 2023  error.log
-rw-r--r-- 1 root root 22G  May 7  05:12 other_vhosts_access.log  

 

3. SSH to controller VM and change to root user 

4. Run the following command :  

#docker inspect -f $'{{.Name}}\t{{.GraphDriver.Data.MergedDir}}' $(docker ps -aq) | grep e7fe54eb13417c94567708492e62a3aa7fc5dc311cd300e3ee42ae7dd6d5a621

5. The output should be something as below : 

/k8s_placement-api_placement-api-5894cb9b97-c2dkl_openstack_93342801-f296-45b3-873b-4786e17edea8_0    /var/lib/docker/overlay2/3f85e033a76be3f65fe2fdaf8d2f97eb240c100568b79b36668c4108397a321b/merged

6. This means the file belongs to Pod placement-api-5894cb9b97-c2dkl

7. You can delete the Pod using cmd below : 

# osctl delete po placement-api-5894cb9b97-c2dkl

8. Post deletion verify the disk space "df -h"