Stopping all reports and extractions. Drive /home is 94% full

Products

Security Analytics Security Analytics - VA

Issue/Introduction

This error can appear on either the Central Manager or on any individual sensor. The percentage can vary, but if the /home partition ever reaches higher than 90% utilization, this error will appear and reports, extractions and pcaps will be halted.

Cause

A large amount of exported pcaps or extractions were likely exported, which can require terabytes of disk space. When you request PCAP files from multiple sensors with few filters over an extended timespan, it will first pull all PCAPs and then store them locally in the /home/apache/pcaps directory. It will then zip up all the PCAPs into one large zip file before presenting the browser the option to download the file. These PCAP files will automatically be deleted after they have been sitting for 72 hours. They are saved for a few hours before they are deleted. In the meantime, the files are held to allow the user to download them to their desktop. This may hold terabytes of space in /home.

Resolution

Typically this message is more prevalent on a Central Manager but can happen on an individual sensor. Run the following command to see which directory is taking up the most space: du -h /home/apache, where the files are stored.

If you run: ls -l /home/apache/pcaps and you see files there that are older than 72 hours, then you may need to delete them manually.

Check the amount of space held by the PCAPs with du -sh /home/apache/pcaps. This returns how much space the PCAPs directory is using in gigabytes. You can also check how much space is used in /home with df -h /home. Is the space Used below 80%?

You can delete all files in the /home/apache/pcaps folder if you choose. The only problem this may cause is if there are active users who are trying to download PCAPs or if there's an active API that is creating PCAPs. If the PCAP files are manually deleted, the user may need to initiate the PCAP download request again.

To manually delete the PCAP files, run the following commands as root:

killall php
systemctl restart httpd
rm -rf /home/apache/pcaps/*

There is another feature, if improperly used can consume large amounts of disk space when files are retrieved from storage, extractions. For example, if someone extracts 24 hours of files without a filter it may use all of the /home filesystem. The files are saved to /home/apache/artifacts. You can check the disk space consumed for the files with du -sh /home/apache/artifacts. This directory can take much more time to cleanup due to the quantity of possible files.

There is a command which will delete the extractions for you. The command is scm extractions delete. After verifying this is what you want to do, it will delete ALL extractions, recent or not, saved or not. This may take 30 minutes or more for a large amount of files extracted. Check the results with du -sh /home/apache/artifacts.

If df -h /home indicates that /home has less than 80% of the space Used, than you can clear the pink banner by selecting the X to the far right side, in the Web UI.

The pink banner may need to be cleared from the command line. If the "X" does not exist in older versions, run the following commandas root: scm db clear_panic

If after running the scm command you see: "Failed to clear UI panic flag", this means that the flag is not active and you should not see the pink banner in the web UI.

At this point, the system will allow the users to save PCAPs, extractions, and reports.