"Could not restore the PGSQL database." error when upgrading from Cloud Director Availability 4.0.x

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Upgrading from Cloud Director Availability 4.0.x fails, and in the /var/log/upgrade.log file on the impacted appliance you see messages similar to:

Restoring PostgreSQL database(s) after upgrade...
('/usr/bin/sudo', '-u', 'h4postgres', '/usr/bin/bash', '-c', "/opt/vmware/vpostgres/current/bin/initdb -D '/opt/vmware/h4/db' --no-locale -E UTF8")
Job for h4postgresql.service failed because the control process exited with error code.
See "systemctl status h4postgresql.service" and "journalctl -xe" for details.
psql.bin: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

gzip: stdout: Broken pipe
 
COULD NOT RESTORE THE PGSQL DATABASE. CONTACT SUPPORT IMMEDIATELY!

• This issue can occur if you had previously upgraded from vCloud Availability 3.x to Cloud Director Availability 4.0.x.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue occurs when there is a database backup file, /opt/vmware/h4/all.sql.gz, from a previous upgrade still present on the appliance when you initiate an upgrade from Cloud Director Availability 4.0.x.

Resolution

Before upgrading a Cloud Director Availability instance that was previously upgraded from vCloud Availability 3.x, validate the state of the database on each appliance.

1. SSH to the Cloud Director Availability appliance you wish to upgrade and log in as the root user.
2. Run the following database queries for the corresponding appliance type and verify each command returns a count of 0.

Cloud Replication Management Appliance:

/opt/vmware/vpostgres/current/bin/psql -U h4postgres h4manager -c "select count(*) from pg_largeobject;"
/opt/vmware/vpostgres/current/bin/psql -U h4postgres h4cloud -c "select count(*) from pg_largeobject;"

 

Cloud Replicator Appliance:

/opt/vmware/vpostgres/current/bin/psql -U h4postgres h4replicator -c "select count(*) from pg_largeobject;"

 

Cloud Tunnel Appliance:

/opt/vmware/vpostgres/current/bin/psql -U h4postgres h4tunnel -c "select count(*) from pg_largeobject;"

3. Confirm the version(s) of the installed VMware Postgres components and that there are no instances of 9.x present:

rpm -qa | grep VMware-Postgres

If the database queries have return a count greater than 0 or you have found VMware Postgres 9.x components installed on the appliances, contact VMware Support and note this Article ID (315039) in the problem description. For more information, see Creating and managing Broadcom support cases (142884).

If the database queries return a count of 0 and there are no VMware Postgres 9.x components installed on the appliances, then the appliances are in a healthy pre-upgrade state. To progress with the upgrade, perform the following steps:

1. Take a backup or snapshot of each appliance in the event you continue to experience upgrade issues.
2. SSH to the Cloud Director Availability appliance you wish to upgrade and log in as the root user.
3. Move the all.sql.gz file to another directory:

mv /opt/vmware/h4/all.sql.gz /root/

4. Upgrade the appliance as per the Upgrading in the Cloud Director site section of the Cloud Director Availability documentation.