Operations Manager UI fails in 500 with the error "could not connect to server: No such file or directory" because no space left on device for PostgreSQL

Products

• Operations Manager

Issue/Introduction

Operations Manager UI fails in 500 with the error "could not connect to server: No such file or directory" in the production.log file for the Ops Manager VM:

E, [2021-07-26T08:14:11.693279 #732] ERROR -- :         Exception caught via rescue_from (code 500):
        (See app/controllers/application_controller.rb)
        could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Manual connection to PostgreSQL via "sudo su - tempest-web -s /bin/bash -c 'psql tempest_production'" fails with:

psql: error: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Check "netstat -ltn", port 5432 of the host is not listened:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.1:8005          0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN        
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN

Following error is observed in postgresql log under /var/log/postgresql/ directory:

2021-07-24 14:22:38.029 UTC [18651] tempest-web@tempest_production ERROR:  could not extend file "base/16387/17307": No space left on device
2021-07-24 14:22:38.029 UTC [18651] tempest-web@tempest_production HINT:  Check free disk space.

Resolution

Ops Manager was running out of space, causing PostgreSQL to fail.

Clear up unneeded files, such as BBR backups on Ops Manager VM. Then restart the Ops Manager VM by either of the following two ways:

• Restart the Ops Manager VM within the IaaS.
• Remoting into the VM via ssh and running sudo reboot.