Error "Internal Server Error" on Security Analytics (Solera) login page or from the widgets on the summary page

book

Article ID: 168502

calendar_today

Updated On:

Products

Security Analytics

Issue/Introduction

The system is not accessible via web console giving error "Internal Server Error".  This is because the postgres database is not able to start correctly.
 

Cause

The most frequent cause of “internal server error” messages is that one of the postgres database is not running.  This is usually due to the /var partition being 100% full.

You can  confirm that the partition is full with the CSR and looking at the "df" file or by running the command "df -h".

Sample output of "df -h"
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda4             4.9G  1.9G  2.8G  41% /
/dev/sda6             2.0G  290M  1.6G  16% /gui
/dev/sda5             4.9G  138M  4.5G   3% /ds
/dev/sda2              68G  68G     0G 100% /var
/dev/sda1             1.5G   12M  1.4G   1% /boot
tmpfs                  63G   12M   63G   1% /dev/shm
/dev/sda3             2.7T  470M  2.5T   1% /home
/dev/sdc1              61T   61T     0 100% /pfs
/dev/sdb1             9.6T  3.0G  9.1T   1% /var/lib/solera/meta1
/dev/sdb2             9.6T  3.0G  9.1T   1% /var/lib/solera/meta2
gaugefs                61T   61T     0 100% /etc/solera/flows

Resolution

Log in to the CLI as root and confirm whether the postgres database is up and running:

service postgresql status

If the database is not running, try to start the database by running:

service postgresql start

If the database cannot start, you need to find out what is taking up all of the space.  Use the following commands to gather data:

du -sh /var/log/*
du -sh /var/lib/pgsql/*
du -sh /var/log/audit/*
du -sh /var/log/csr-history/*


Often you can delete old log files to free up enough space to get the database running again.  Directories that are typically safe to clean out are /var/log/audit/ and /var/log/csr-history/  If the space is being consumed in these log directories, after deleting the logs, try to start up the postgres database and attempt to log in again.

If all of the space is located in the /var/lib/pgsql directory structure, you will need to contact Blue Coat Support in order to receive direction on what tables in the database can be manually cleaned up.

If space is not an issue, look at permissions on /tmp.  They should look like:

ls -ld /tmp   drwxrwxrwt 7 root root 4096 2017-02-23 14:24 /tmp

To set /tmp permission properly run chmod a+rwxt /tmp and check permissions again.

If either of these were the problem and have been resolved, running service postgresql start will bring up postgres.  Then attempt to view the UI login page from the browser.

To get more detail as to why postgres is not starting, you can run:    bash -x /etc/init.d/postgresql start

Collect the output and send it to support and we can assist where needed.