/pfs and and /var/lib/solera/meta filesystems showing very full

Products

• Security Analytics
• Security Analytics - VA

Issue/Introduction

Several file systems on a Security Analytics appliance will always show higher utilization that others.  This is not necessarily a bad thing and shows that the appliance is working as designed.

Resolution

On a Security Analytics capture appliance (not a Central Manager), two file systems will always be 100% full or close to it: "/pfs" and "/etc/solera/flows".  These two file systems will normally match exactly.

[root@hostname ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda4             4.9G  3.0G  1.6G  66% /
/dev/sda6             2.0G  458M  1.4G  25% /gui
/dev/sda2              68G  5.0G   59G   8% /var
/dev/sda1             1.5G   41M  1.4G   3% /boot
/dev/sda5             4.9G  1.5G  3.2G  32% /ds
tmpfs                  63G  228M   63G   1% /dev/shm
/dev/sda3             2.7T  3.2G  2.5T   1% /home
/dev/sdc1              21T   21T     0 100% /pfs
/dev/sdb1             9.6T  7.9T  1.6T  83% /var/lib/solera/meta1
/dev/sdb2             9.6T  7.9T  1.6T  83% /var/lib/solera/meta2
gaugefs                21T   21T     0 100% /etc/solera/flows

Both /pfs and /etc/solera/flows are virtual filesystems where all space is pre-allocated. They will always show as 100% (or close to 100%) full when mounted.  As the data is captured, the /pfs file system uses FIFO (first in first out) for managing new traffic.  The oldest data will always be overwritten first.

If /pfs is not listed in the output of "df" on a capture appliance, the system may not be licensed. As root, run "service solera status" to determine why /pfs is not mounted. If the capture filesystem is not running, that command should return "unlicensed". If it returns any other status, or if assistance with licensing is needed, contact technical support.

If /etc/solera/flows is not listed in the output of "df" and the appliance is not a Central Manager, the gaugefs service is stopped. Contact technical support for assistance.

The index file systems (/var/lib/solera/meta*) are managed by the indexing services. Once they reach 80% full, the indexer will, as needed, remove old index data in order to keep them less than or equal to 83% full. When these filesystems grow to >83%, the indexing services will remove enough old data to lower the utilization back to 80%.

If an index filesystem shows >83% full for sustained periods of time, contact technical support for assistance.