/var partition is filling up on Security Analytics and how to clean up disk space

book

Article ID: 168963

calendar_today

Updated On:

Products

Security Analytics

Issue/Introduction

/var partition is filling up on Security Analytics.  This could be tables in the postgres database. You can check with the command "scm db summary dsweb size". The last line is the size of the largest table.

This may also be caused by audit logs growing very large and not being trimmed.  This is seen with lsof | grep delete | grep audit.log 

 

Cause

The auditd daemon is monitoring a highly active directory, which has minimal security exposure.

Resolution

The main postgres database resides in /var/lib/pgsql.  If the database has filled up all available space in /var, it will cause issues with authentication and other basic functions.  Typically the /var partition is 68GB in size.  It is possible that there are log files that are also filling up the partition.  It's good to check these commands to see how large certain directories are:

du -sh /var/log
du -sh /var/lib/pgsql


If the space is being taken up by log files such as the /var/log/messages file or audit files in /var/log/audit, you can temporarily move them to the /home directory if needed to free up space.  

NOTE:  This procedure is not recoverable.  Once you truncate a table in the database, you cannot retrieve it.  Truncating the wrong table can render your appliance useless.  Unless you are already familiar with these different tables, please contact Technical Support to validate that truncating tables is the right solution.

If the largest table is report_items, run: 

su - postgres 
psql -d dsweb 
truncate table report_items cascade;
\q 
exit 
scm db summary dsweb size 
df -h 



Note: If the meta_info table is the largest, run the command "truncate table meta_info cascade;". 

Another cause for /var filling is the audit.log collecting the changes made in Anomaly Detection.  Only perform these steps if you are running Security Analytics version 7.2.1 or earlier.  This setting has been fixed in Security Analytics version 7.2.2 and greater.

  1. Edit the /etc/audit/audit.rules file and find the following line:

-A exit,never -F arch=b64 -F dir=/var/spool/prelert

  1. Add the following line just below (if it doesn't already exist): 

-A exit,never -F arch=b64 -F dir=/opt/prelert/prelert_home/cots/elasticsearch

  1. Save and exit this file.

To enable proper audit.log rotation:

  1. edit /etc/audit/auditd.conf
  2. Change:  log_file = /var/log/audit/audit.log to  log_file = /var/tmp/audit_unused
  3. Then restart auditd with service auditd restart.