Drive / (root) is 100% full in Security Analytics

Products

Security Analytics

Issue/Introduction

Pink banner in the Security Analytics GUI shows: "Stopping all reports and extractions. Drive / is 100% full. Metric=bytes, Threshold=90

You may see an "Internal Error" when trying to log in to the web interface.

Cause

The / partition or the root partition must have free space in order for the system to behave properly. Temporary files are constantly being added and updated. If root is full, it can cause all sorts of side effects such as not being able to log in.

Resolution

There are certain directories that should be checked to see if there are large files taking up space. This exercise is best done when you can compare the command output with a healthy appliance.

First confirm that root is full using the 'df -h' command at the command line while logged in as root.

You should see something like this (sizes will vary depending on the OS version you are running):

Filesystem             Size  Used Avail Use% Mounted on
/dev/sda4              7.6G  7.6G     0 100% /

You can use this command to check the sizes of each directory beneath ' / ' :

du -h --exclude=/pfs --exclude=/etc/solera/flows --exclude=/var --exclude=/home --exclude=/ds --exclude=/gui --exclude=/run --exclude=/dev --exclude=/proc --exclude=/boot / | sort -h

This will produce a long list of directories sorted with the largest directories at the bottom. Anything with sizes in the GB range should be checked.

Normal culprits where large files may exist are in the following directories:

/tmp (look for old csr. files or large files and delete them)

/opt/prelert/prelert_home/logs/elasticsearch (delete any large files)

/usr/share/jsunpackn/temp/ (delete any large logs)

/root (look for a file called 'dead letter' and if it is large, delete it)

/ (look for a large file called MegaSAS.log and delete it. Upgrade to SA version 8.1.2 or greater to prevent this file from growing large again)

Another command to use to find the largest files in the / (root) filesystem is:

find / -mount -type f -printf '%s %p\n'| sort -nr | head --lines=30

You may change / to /var for the /var filesystem, if it is full.

Once large files have been deleted, the appliance may need to be rebooted for the disk cleanup process to finalize. If you are nervous about deleting files, contact technical support for guidance.

It is possible that even after disk space has been cleaned up, the pink banner with the error message may still remain. You can either click the "X" on the message in the GUI to dismiss it or you can clear the banner from the command line. If the "X" does not exist, log in to the CLI as root and run the following command: scm db clear_panic