SDDC Manager UI fails with error '502 Bad Gateway' due to Postgres service not starting

Products

• VMware SDDC Manager

Issue/Introduction

• SDDC Manager UI shows error message '502 Bad Gateway' immediately after attempting to access it similar to as shown below:

• Operations manager, Domain manager, commonsvcs logs are not being generated and nginx fails to start
• SDDC Manager Appliance VM-console shows "[FAILED] Failed to start Postgres" message
• Running the following command to check the status of postgres service shows "could not translate host name 'localhost'". This means the SDDC Manager could not resolve itself.

 # systemctl status postgres
* postgres.service - Postgres
     Loaded: loaded (/etc/systemd/system/postgres.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Day YYYY-MM-DD HH:MM:SS UTC;
    Process: 910 ExecStartPre=/usr/bin/sh -c /opt/vmware/vcf/commonsvcs/scripts/postgres/postgres-upgrade.sh /usr/pgsql/15/bin/ 15 (code=exited, status=0/SUCCESS)
    Process: 920 ExecStart=/usr/pgsql/15/bin/pg_ctl start -D ${PGDATA} -s -w -t 270 (code=exited, status=1/FAILURE)
MM DD HH:MM:SS <sddc_manager_fqdn> pg_ctl[940]: YYYY-MM-DD HH:MM:SS UTC [940] LOG:  starting PostgreSQL 15.12 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 10.5.0, 64-bit
MM DD HH:MM:SS <sddc_manager_fqdn> pg_ctl[940]: YYYY-MM-DD HH:MM:SS UTC [940] LOG:  could not translate host name "localhost", service "5432" to address: Temporary failure in name resolution
MM DD HH:MM:SS <sddc_manager_fqdn> pg_ctl[940]: YYYY-MM-DD HH:MM:SS UTC [940] WARNING:  could not create listen socket for "localhost"
MM DD HH:MM:SS <sddc_manager_fqdn> pg_ctl[940]: YYYY-MM-DD HH:MM:SS UTC [940] FATAL:  could not create any TCP/IP sockets
MM DD HH:MM:SS <sddc_manager_fqdn> pg_ctl[940]: YYYY-MM-DD HH:MM:SS UTC [940] LOG:  database system is shut down
MM DD HH:MM:SS <sddc_manager_fqdn> pg_ctl[920]: pg_ctl: could not start server
MM DD HH:MM:SS <sddc_manager_fqdn> pg_ctl[920]: Examine the log output.
MM DD HH:MM:SS <sddc_manager_fqdn> systemd[1]: postgres.service: Control process exited, code=exited, status=1/FAILURE
MM DD HH:MM:SS <sddc_manager_fqdn> systemd[1]: postgres.service: Failed with result 'exit-code'.
MM DD HH:MM:SS <sddc_manager_fqdn> systemd[1]: Failed to start Postgres.

• The loopback address resolution in the "/etc/hosts" file is commented out.

# Begin /etc/hosts (network card version)

# End /etc/hosts (network card version)
# VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
# 127.0.0.1 <sddc_manager_fqdn> <sddc_manager_hostname> localhost
# :: 1 <sddc_manager_fqdn> <sddc_manager_hostname> localhost ipv6-localhost ipv6-loopback
# VAMI EDIT END

Environment

• VCF 5.x
• VCF 9.x

Cause

The Postgres service failed to initialize due to a commented-out loopback address entry in the /etc/hosts file. This configuration prevented the service from successfully resolving its local hostname (localhost), which is crucial for its internal operations and startup.

Resolution

1. Take backup of the /etc/hosts file:

cp /etc/hosts /etc/hosts.backup

2. Edit the /etc/hosts file using vi editor:

vi /etc/hosts

3. Edit the hosts file and remove the # for ipv4 and ipv6 loopback address resolution as an example below:

# Begin /etc/hosts (network card version)

# End /etc/hosts (network card version)
# VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 <sddc_manager_fqdn> <sddc_manager_hostname> localhost
:: 1 <sddc_manager_fqdn> <sddc_manager_hostname> localhost ipv6-localhost ipv6-loopback
# VAMI EDIT END

4. Save the file using 'Esc + :wq!' and press Enter to exit vi editor.

5. Verify the changes:

cat /etc/hosts

6. Restart SDDC Manager Appliance.