High disk usage for application

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

You may face application instance disk usage reaching 100% which causes the application instance to fail.
Customers need to restart the app frequently to reduce disk space 

This may cause due to unreleased files by the application. When the application process file and deletes a file, the file is deleted successfully but the size of the filesystem does not reflect the change.  As the process still holds the file.

To verify what causes disk usage

Log in to the app container as the root KB

Verify the list of open files using lsof

Obtain a list of deleted files that are still held open by applications:

$ lsof | egrep "deleted|COMMAND"
COMMAND       PID    TID TASKCMD     USER   FD  TYPE  DEVICE    SIZE/OFF      NODE NAME
ora         25575   8194 oracle    oracle   33   REG   65,65  4294983680  31014933 /oradata/DATAPRE/file.dbf (deleted)

The lsof output shows the process with pid 25575 has kept file /oradata/DATAPRE/file.dbf open with file descriptor (fd) number 33.

To identify the used file size (in blocks), use the command below:

lsof -Fn -Fs |grep -B1 -i deleted | grep ^s  | cut -c 2- | awk '{s+=$1} END {print s}'

After the file and space used by files have been identified, free the space by shutting down/killing the affected process. which keeps the files open. This PID may be a sub-thread of the original app or the main thread of the app

Note: Killing PID may restart your app instance. if your app uses main thread to process the files
Note: Free the space by shutting down/killing the affected process is a temporary solution which is the same as restarting app

Environment

Product Version: 2.11

Resolution

Root Cause

On Linux or Unix systems, deleting a file via rm or through a file manager application will unlink the file from the file system's directory structure; however, if the file is still open (in use by a running process) it will still be accessible to this process and will continue to occupy space on disk. Therefore such processes may need to be restarted before that file's space will be cleared up on the filesystem.

It is recommended to create an application with multithread if you want to process files. Close the sub-thread after file processing/deleting of the file is complete.