High system (root) disk usage on Diego Cell

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Application Service

Issue/Introduction

You may observe that a Diego Cell vm is running out (or has ran out) of disk space. You want to identify what is consuming the space first so you can then troubleshoot the cause. 

*This article will concentrate on how to identify the source of what is consuming the system (root) disk space.

Cause

Excessive disk usage is often caused by a process or job generating large or numerous files such as logs core dump or temp files.  This can often lead to the root filesystem becoming full.

Resolution

1. ssh onto affected diego cell vm:

bosh -d cf-<DEPLOYMENT-NAME> ssh diego_cell/<VM-NAME>

2. Become sudo

sudo -i

3. Check current disk usage:

df -h

3a. Check to see if root filesystem is 100% used (or close).

# Example output from "df -h": 

Filesystem     Size  Used  Avail  Use%    Mounted on
/dev/sda2      4.8G  4.6G     0    100%        /

4. Run below command(s) to identify which directories are using the most space for the "/" root directory.

# First run below command:
du -h / --max-depth=1 | sort -rh | head -n 10

# Then look to see if any directory/folder is larger than expected and explore further. Example below:

# for this example "/var" is larger than anticipated, then we explore it further:
du -h /var --max-depth=1 | sort -rh | head -n 10

5.  Investigate further to see what type of files are filling up the system disk space. For our example in #4, we identified the /var directory. Now lets have closer look at the log files to see where/what job or process this coming from:

ls -lh /var/log | sort -k 5 -rh | head

* Now we should have see the biggest consumers/files of the system disk space.

Additional Information

Note: Deleting files may cause system issues.  Always verify actions and proceed with caution.