/var is full on Security Analytics

Products

• Security Analytics
• Security Analytics - VA

Issue/Introduction

If the /var directory is full on a Security Analytics appliance, it can cause adverse affects, mainly preventing the postgres database from function correctly.  This can prevent abnormal issues with the browser GUI.

Cause

Audit logs are taking up all the available space.  The audit logs record each time a file is read or written or otherwise modified.  This can be a security requirement at a site.  This requirement is not typical.  Disabling audit logging will reduce the traffic to the primary operating system disks and reduce the possibilities of filling the /var filesystem.  These files are typically in /var/log/audit.

Resolution

Update /etc/audit/audit.rules as root and restart auditd.  

Find the section similar to:

# /dev/shm/var/lib/solera
-A exit,never -F arch=b64 -F dir=/dev/shm/var/lib/solera

# /pfs
-A exit,never -F arch=b64 -F dir=/pfs

Add the following lines

# Exclude all files in /var/lib/solera
-A exit,never -F arch=b64 -F dir=/var/lib/solera

Restart auditd and syslog-ng with:

service auditd restart;service syslog-ng restart

The /var/log/audit/audit.log will be rotated and start with an empty file.

Also, when looking for files to clear to free up space, look in /var/log/spool/prelert/ and /var/spool/prelert/elasticsearch/data/prelert/nodes/0/indices for directories like 20191126080448-00001 which have not been modified in six months.  These dated directories will be databases used for Anomaly Detection.  The old directories are no longer used.

Additional Information

See the following KB article for more causes for /var filling up: /var partition is filling up or is 100% utilized