SDDC Manager UI is inaccessible. Error: 502 Bad Gateway due to the SDDC Database being down.

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms:  

• The SDDC database is down after performing an upgrade of the SDDC Manager.
• Attempts to start the database manually fail
• All of the SDDC services are in a dead state. 
• The following error is found in the journalctl logs. 
  
  journalctl -xe | less 

MMM DD HH:MM:SS #########.#########.#### pg_ctl[527944]: MMM DD HH:MM:SS [527944] LOG: starting PostgreSQL 15.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 10.5.0, 64-bit
MMM DD HH:MM:SS #########.#########.#### pg_ctl[527944]: MMM DD HH:MM:SS [527944] LOG: listening on IPv4 address "127.0.0.1", port 5432
MMM DD HH:MM:SS #########.#########.#### pg_ctl[527944]: MMM DD HH:MM:SS [527944] FATAL: could not open lock file "/home/postgresql/.s.PGSQL.####.lock": Permission denied
MMM DD HH:MM:SS #########.#########.#### pg_ctl[527944]: MMM DD HH:MM:SS [527944] LOG: database system is shut down
MMM DD HH:MM:SS #########.#########.#### pg_ctl[527942]: pg_ctl: could not start server
MMM DD HH:MM:SS #########.#########.#### pg_ctl[527942]: Examine the log output.
MMM DD HH:MM:SS #########.#########.#### systemd[1]: postgres.service: Control process exited, code=exited, status=1/FAILURE

Environment

SDDC Manager 5.2.1

Cause

The /home/postgresql directory was owned by the vcf user instead of the postgres user, which prevented the PostgreSQL service from starting.

Resolution

• Take a snapshot of the SDDC Manager VM
• Check the owner of the /home/postgresql directory: ls -ltrah 
• Change the ownership of the directory: chown postgres -R postgresql/
• Start the SDDC database: systemctl start postgres
• Restart all the SDDC Manager services: /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh