/var partition is filling up or is 100% utilized

Products

Security Analytics Security Analytics - VA

Issue/Introduction

The /var partition can fill up if there are certain files growing in size.

This error may appear in the GUI: "Stopping all reports and extractions. Drive /var is xx% full. Metric=bytes, Threshold=90".

Cause

This can be caused by large tables in the postgres database.

This can also be caused by audit logs growing very large and not being trimmed. This is seen with lsof | grep delete | grep audit.log

The auditd daemon is monitoring a highly active directory, which has minimal security exposure. This can also be caused by very large files in the /var/log/prelert directory (usually in the elasticsearch subdirectory).

Resolution

Outdated databases in prelert (Anomaly Detection)

Sometimes there may be large outdated databases in /var/spool/prelert/elasticsearch/data/prelert/nodes/0/indices. There may be old prelert databases used for anomaly detection which have been replaced with newer releases. Look for directories where the name is a date. For example, /var/spool/prelert/elasticsearch/data/prelert/nodes/0/indices/20191202111246-00001.

Look at the modification date with ls -ld /var/spool/prelert/elasticsearch/data/prelert/nodes/0/indices/*

Determine how much space these directories are holding with du -sh /var/spool/prelert/elasticsearch/data/prelert/nodes/0/indices/* You may safely delete any directories which are over six months old.

Large files in /var/log/prelert directory

If the large files are in the /var/log/prelert directory, confirm whether Anomaly Detection is enabled by going to the Settings > Data Enrichment > Data Enrichment Profiles section. If Anomaly Detection is enabled, the profile selected will be "Full Data Enrichment with Anomaly detection". This was the default setting for several versions of Security Analytics and is a feature that you may not realize is enabled. Disable Anomaly Detection by choosing the "Full Data Enrichment (No Anomaly Detection)" profile. It should automatically clean up the files and logs in /var/log/prelert/elasticsearch, but if it does not, the files can be manually deleted.

Run: du -sh /var/log/prelert/engine_api/stderr.log

If stderr.log is larger than a few megabytes, it can be removed. This file can be emptied by running: cp /dev/null /var/log/prelert/engine_api/stderr.log

Large tables in the postgres database

The main postgres database resides in /var/lib/pgsql. If the database has filled up all available space in /var, it will cause issues with authentication and other basic functions. Typically the /var partition is 68GB in size. It is possible that there are log files that are also filling up the partition. It's good to check these commands to see how large certain directories are:

du -sh /var/log
du -sh /var/lib/pgsql

If the space is being taken up by log files such as the /var/log/messages file or audit files in /var/log/audit, you can temporarily move them to the /home directory if needed to free up space.

It is possible that the postgres database is not running due to the limited space available. Postgres requires space to utilize temporary files.

NOTE: postgres commands will not work unless the postgres service is running.

Confirm that the postgres service is running: systemctl status postgresql

Look for the line:

Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-08-03 20:59:37 MDT; 7 months 17 days ago

If it is not running, restart postgres: systemctl restart postgresql

Use the following command to see which tables are taking up the most space in the postgres database: scm db summary dsweb size

NOTE: You can clean up tables in the database, but it is recommended you check with technical support before you delete any of these tables on your own. This procedure is not recoverable. Once tables are truncated in the database, they are unretrievable. Truncating the wrong table can render your appliance inoperable.

If the largest table is 'report_items', to clean up the table, you would run (as root):

#
su - postgres
psql -d dsweb
truncate table report_items cascade;
\q
exit
#

Confirm psql and /var allocation again using these commands:

scm db summary dsweb size
df -h

audit.log filling with Anomaly Detection data

Another cause for /var filling is the audit.log collecting the changes made in Anomaly Detection. Only perform these steps if you are running Security Analytics version 7.2.1 or earlier. This setting has been fixed in Security Analytics version 7.2.2 and greater.

Edit the /etc/audit/audit.rules file and find the following line:

-A exit,never -F arch=b64 -F dir=/var/spool/prelert

Add the following line just below (if it doesn't already exist):

-A exit,never -F arch=b64 -F dir=/opt/prelert/prelert_home/cots/elasticsearch

Save and exit this file.