SDDC Manager UI inaccessible with error "502 Bad Gateway" due to postgres service.

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Unable to access the SDDC Manager user interface (UI). Attempts to access the page result in below error message.
  • 502 Bad Gateway (nginx)
    
    
• Access validation to the SDDC Manager UI fails explicitly due to the underlying postgres.service failing to start.
  
  
• All of the SDDC services are in a dead state. 
  
  
• The following error is found in the journalctl logs. 
  • journalctl -xe
    • Month DD hh:mm:yy <sddc_manager_fqdn> systemd[1]: postgres.service: Control process exited, code=exited, status=1/FAILURE
Subject: Unit process exited
Defined-By: systemd
Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
An ExecStart= process belonging to unit postgres.service has exited.

The process' exit code is 'exited' and its exit status is 1.
Month DD hh:mm:yy <sddc_manager_fqdn> systemd[1]: postgres.service: Failed with result 'exit-code'.
Subject: Unit failed
Defined-By: systemd
Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

The unit postgres.service has entered the 'failed' state with result 'exit-code'
Month DD hh:mm:yy <sddc_manager_fqdn> systemd[1]: Failed to start Postgres.
Subject: A start job for unit postgres. service has failed
Defined-By: systemd
Support: https://lists. freedesktop. org/mailman/listinfo/systemd-devel

A start job for unit postgres.service has finished with a failure.
The job identifier is ##### and the job result is failed.

• The /var/log/postgres/postgres.log file reveals the following underlying fatal error:
  
  • YYYY-MM-DD hh:mm:ss UTC 6a####b3.c#8 0  3304 3LOG: starting PostgreSQL 15.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 10.5.0, 64-bit
YYYY-MM-DD hh:mm:ss UTC 6a####b3.c#8 0  3304 4LOG: listening on IPv4 address "127.0.0.1", port 5432
YYYY-MM-DD hh:mm:ss UTC 6a####b3.c#8 0  3304 5FATAL: bogus data in lock file "/home/postgresql/.s.PGSQL.5432. lock": ""
YYYY-MM-DD hh:mm:ss UTC 6a####b3.c#8 0  3304 6LOG: database system is shut down

Environment

• VMware SDDC Manager 5.2.X

Cause

• An empty or corrupted socket lock file (.s.PGSQL.5432.lock) exists in the socket directory. When the database engine starts up, it reads this file to find the process ID of an active instance; since the file contains empty data (""), the parser throws a FATAL exception and terminates the process immediately.

Resolution

• Take a snapshot of the SDDC Manager before making any changes. 
• Log in to the SDDC Manager via SSH as vcf and switch to root.
• Verify the corrupt lock file properties:

  • ls -la /home/postgresql/.s.PGSQL.5432.lock

• Move the corrupt lock file out of the directory to a backup name:
  

  • mv /home/postgresql/.s.PGSQL.5432.lock /home/postgresql/.s.PGSQL.5432.lock.bak

• Start the PostgreSQL database service manually:
  

  • systemctl start postgres

• Confirm that the service successfully transitions to an active running state:
  

  • systemctl status postgres

• Restart the dependent SDDC Manager operations services:
  

  • /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh