SDDC Manager 5.2.1 upgrade fails during reboot stage.

Products

• VMware Cloud Foundation 5.x

Issue/Introduction

• SDDC Manager UI hangs on the Refresh Custom Certificates stage.
  
  
  
  
  
• All of the SDDC services are in an dead state. 
• The following error is found in the journalctl logs. 
  
  journalctl -xe | less

2563739 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + POSTGRES_OLD_UID=1002
2563740 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + '[' 0 -ne 0 ']'
2563741 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + '[' 1002 -le 999 ']'
2563742 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + echo 'User '\''postgres'\'' is not a system user. Converting to system user.'
2563743 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: User 'postgres' is not a system user. Converting to system user.
2563744 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + TEMP_SYSTEM_USER=postgres_system
2563745 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + systemctl stop postgres.service
2563864 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + useradd -r postgres_system --shell=/sbin/nologin
2564124 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + POSTGRES_NEW_SYSTEM_USER_UID=985
2564125 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + userdel postgres_system
2564229 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + usermod -u 985 postgres
2564248 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx sh[909]: + find / -uid 1002 -exec chown postgres '{}' ';'
2564761 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx bash[2657]: creating operationsmanager db
2564762 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx bash[2653]: creating database platform
2564763 MMM DD HH:MM:SS xxxxxxxxxxxxxxx.xxx.xxxxx.xx bash[2677]: psql: error: connection to server at "localhost" (127.0.0.1), 

Environment

SDDC Manager 5.2.1

Cause

• This issue arises because the postgres user is not set up as a system user. This configuration affects deployments of SDDC Manager initially deployed on version 4.4.x or earlier

Resolution

Workaround:

1. SSH into the SDDC Manager Appliance with the vcf user and then elevate to root with su.

2. Restart the postgres service
systemctl restart postgres

3. Log back into the SDDC UI and finish the SDDC Upgrade. 

 

Proactive Workaround: (Prior to running the 5.2.1 SDDC Upgrade)

1. SSH into the SDDC Manager Appliance with the vcf user and then elevate to root with su.

2. Copy the postgres-system-user.sh script from the KB to the /home/vcf/ directory on the SDDC manager. 

3. Set the required permissions on the file. 
chmod +x postgres-system-user.sh

4. Run the script. 
./postgres-system-user.sh

5. Initiate the 5.2.1 SDDC Upgrade

Additional Information