Insufficient Disk Space - The /var partition is 85% full. Metric=bytes. Threshold=80

Products

• Security Analytics
• Security Analytics - VA

Issue/Introduction

This may show up in the notifications or audit log section of the Security Analytics UI.  

This can also be confirmed by going to the CLI and running the following command as root:   df -h

Look for the /var partition and confirm the Use%.  

This message may appear in the /var/log/messages file as well:

hostname php[72054]: snlog: sn="##:##:##:##:##:##" id="DS" m="69" c="1" event="EVENT_INSUFFICIENT_DISK_SPACE" category="SYSTEM" ip="##.##.##.##" model="R640xl" msg="logmsg=model.sys_log::options.event.audit.size_drive_warning, partition=/var, percentage=85%, metric=bytes, threshold=80"

Cause

This can be caused a variety of reasons.  In older versions, it was caused by the audit log filling up the /var partition.  Insufficient Disk Space: /var is xx% full on Security Analytics.

Other reasons could be linked to Anomaly Detection or prelert being enabled.  /var partition is filling up or is 100% utilized

Resolution

Sometimes this message will come and go because of a cron job that will periodically clean up old files in the /var partition.  If you get this message and df -h shows that the Use% of /var is below 80%, the cleanup process is doing its job.  

If you are curious where the large file sizes are coming from (postgres or audit or prelert), run the following command, which will list all directories in the /var partition (excluding the /var/lib/solera/meta directory, which is its own partition).  

Run the following command from the CLI as root: 

du -h /var --exclude=/var/lib/solera/meta* | sort -n -r | sort -h

If you have questions about whether you can delete any of these large files manually, contact technical support.