The SDDC manager upgrade failed after the reboot stage

Products

• VMware SDDC Manager

Issue/Introduction

• During the upgrade of the SDDC manager, the upgrade fails at reboot stage. 
• The SDDC Manager interface displays an Error 502 (Bad Gateway).
• Attempts to restart services via the command line fail
• The PostgreSQL service is in a stopped state. Manual attempts to start the service (systemctl start postgres) result in an immediate failure.
• journal -xe will report the following errors. 

MM DD HH:MM:SS [email protected]  kernel: audit: type=1006 audit(1769503658.930:77): pid=10313 uid=0 subj=unconfined old-auid=4294967295 auid=985 tty=(none) old-ses=4294967295 ses=12 res=1
MM DD HH:MM:SS [email protected] sh[10314]: Performing Consistency Checks
MM DD HH:MM:SS [email protected] sh[10314]: -----------------------------
MM DD HH:MM:SS [email protected] sh[10314]: Checking cluster versions                                   ok
MM DD HH:MM:SS [email protected] sh[10314]: connection to server on socket "/tmp/.s.PGSQL.50432" failed: FATAL:  database "template1" does not exist
MM DD HH:MM:SS [email protected] sh[10314]: DETAIL:  The database subdirectory "base/16400" is missing.
MM DD HH:MM:SS [email protected] sh[10314]: could not connect to source postmaster started with the command:
MM DD HH:MM:SS [email protected] sh[10314]: "/usr/pgsql/13/bin/pg_ctl" -w -l "/data/pgdata15/pg_upgrade_output.d/20260127T084738.940/log/pg_upgrade_server.log" -D "/data/pgdata" -o "-p 50432 -b  -c listen_addresses='' ->
MM DD HH:MM:SS [email protected] sh[10314]: Failure, exiting
MM DD HH:MM:SS [email protected] su[10313]: pam_unix(su:session): session closed for user postgres
MM DD HH:MM:SS [email protected] sh[10275]: + '[' 1 -ne 0 ']'
MM DD HH:MM:SS [email protected] sh[10275]: + echo 'Postgres upgrade has failed, Please contact Engineering for further support'
MM DD HH:MM:SS [email protected] sh[10275]: + echo 'Postgres Upgrade Failed'
MM DD HH:MM:SS [email protected] sh[10275]: Postgres Upgrade Failed
MM DD HH:MM:SS [email protected] sh[10275]: + exit 1
MM DD HH:MM:SS [email protected] systemd-logind[938]: Session c9 logged out. Waiting for processes to exit.
MM DD HH:MM:SS [email protected] systemd[1]: postgres.service: Control process exited, code=exited, status=1/FAILURE

Environment

VCF 5.x 

Cause

This issue is typically caused by missing or corrupted directory structures within the /data/pgdata/ path. In this specific scenario, the directory 16400 is missing from the /data/pgdata/base path because files have been manually modified, moved, or deleted. This results in the pg_upgrade utility reporting a database inconsistency which prevents the SDDC Manager upgrade from completing successfully.

Resolution

To resolve the issue: 

Find the missing directory or files across the SDDC manager., 

• Revert the snapshots of the SDDC manager prior to the upgrade. 
• Login to the SDDC manager using as vcf user and elevate to root. 

su root

• Search for the missing database files/directory using find command. 

find / -iname 16400

• This will result in one of the two outcomes. 
  1. The missing directory/file found to be present at a different path. 
  2. The directory or file is missing 

Scenario 1: Missing directory found to be present at a different path.

• Stop the postgres service 

systemctl stop postgres

• Moved the missing directory from found path to default path '/data/pgdata/base' 
  • For example, if the directory was found at /nfs/vmware/vcf/nfs-mount/16400, then 

mv /nfs/vmware/vcf/nfs-mount/16400 /data/pgdata/base/ 

• Start the postgres service.

 systemctl start postgres

• Restart SDDC manager services. 

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

• Restart the upgrade 

Scenario: 2: The directory or file is missing and not found using the find command. 

• If directory or files are missing from the SDDC manager, contact Broadcom Support for further assistance.