Clear unprocessed temporary files in the backend; disk space is nearly full

Products

• VMware Smart Assurance

Issue/Introduction

Steps to clear unprocessed temporary files in the backend. Disk space is full due to the *.atf and *.mysql files.

Environment

Watch4Net/MnR - 7.x

Cause

When the backend hangs or is busy due to multiple simultaneous database tasks, the temporary file count will accumulate.

Resolution

• Files in the <APG_BASE>/Backends/APG-Backend/<INSTANCE>/tmp/ folder (.atf or .mysql) are normally processed quickly without accumulating.
• However in some cases, when the backend hangs or is busy due to multiple simultaneous database tasks, the temporary file count will accumulate.
• This can lead to a significant reduction in disk space that can exacerbate the file processing issue. 

Steps to force the backend to process temporary files that have not been cleared.

1. Run the following command to determine which 10 directories in Watch4net/MnR are using the allocated disk space:

du -h <APG_BASE>  |sort -k1nr |head -n 10

2. Once it's determined which backend database is using the disk space, navigate to the backend /tmp folder and determine how many files are in the directory. 

ls -1 | wc -l
 

Note: .mysql and .atf files contain the time series data for Watch4net/MnR reports and thus should not be deleted.

3. Navigate to <APG_BASE>/bin/ and attempt to stop the backend in question. If the backend fails to stop, kill the process:

• Find the PID of the backend process in the following file: <APG_BASE>/Backends/APG-Backend/<INSTANCE>/logs/apg-backend-<INSTANCE>.pid
• Execute: kill -9 <PID> 

4. Stop the MySQL database service using the same method as step 3. 
5. Start the MySQL database, and then the backend process:

• <APG_BASE>/bin/manage-modules.sh service start mysql Default
• <APG_BASE>/bin/manage-modules.sh service start backend <INSTANCE>

6. Verify that the temporary file count under the <APG_BASE>/Backends/APG-Backend/<INSTANCE>/tmp/ directory is actively reducing: ls -1 | wc -l
7. Confirm that the import-properties task is completing successfully and without errors on the frontend host. 
8. After a few hours (depending of the number of files) the files will be processed and the filesystem utilization should reduce. 
9. If there are errors in the Webservice gateway logs (<APG_BASE>/Tools/Webservice-Gateway/Default/logs/gateway-*-*.log) pertaining to the backend service that was restarted. Delete the .webservice file and restart the backend service:

• Delete the .webservice file under <APG_BASE>/Backends/APG-Backend/<INSTANCE>/ (Note: This file may be hidden; please use the ls -la command to check for it's existence). 
• Restart the backend service: <APG_BASE>/bin/manage-modules.sh service restart backend <INSTANCE>
• After the temporary files have been processed, the .webservice file will be regenerated by the backend service.