Failed to start VMware Postgres database server after upgrading vSphere Replication appliance

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• After upgrading vSphere Replication appliance, the vSphere Replication Management (VRM) service appears to be stopped in VAMI, and the /opt/vmware/hms/logs/hms-configtool.log file in the appliance contains java.net.ConnectException: Connection refused error messages.
• Unable to reconfigure the replication after upgrade.
• When restarting the vSphere replication VM we can see this error while the VM is booting:
  • Failed to start VMware Postgres database server
  • 

• hms.log have below entries:

  YYYY-MM-DD 12:31:35.762 WARN  com.mchange.v2.resourcepool.BasicResourcePool [C3P0PooledConnectionPoolManager[identityToken->z8kfltbcqn92a01jlqsa|3a66e67e]-HelperThread-#19] (..log.slf4j.Slf4jMLog$Slf4jMLogger$WarnLogger) [] | com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@347f7723 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
  org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
          at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:280) ~[postgresql-jdbc-driver-8.6.0.jar:42.2.5]
         ..
  Caused by: java.net.ConnectException: Connection refused (Connection refused)
  

Environment

VMware vSphere Replication 8.x

Cause

This issue is observed when the embedded DB schema upgrade fails because the vPostgreSQL service was not fully started.

Resolution

1. SSH to the VR appliance

2. Login as admin and su as root 

3. Run the command - chmod 777 /var/log/vmware/vrmsdb/postgresql.log

4. Run this command to upgrade the DB Schema - 

/opt/vmware/hms/bin/hms-configtool -cmd upgrade -configfile /opt/vmware/hms/conf/hms-configuration.xml (Ignore the error)

5. Reboot the appliance 

6. Reconfigure the Appliance.

Additional Information

The vSphere Replication Management service does not start after the upgrade

After you upgrade vSphere Replication, the vSphere Replication Management (VRM) service appears as stopped in the VAMI, and the /opt/vmware/hms/logs/hms-configtool.log file in the virtual appliance contains java.net.ConnectException: Connection refused error messages.

This problem is observed if the upgrade procedure of the embedded DB schema fails because the vPostgreSQL service was not fully started.

Workaround:

1. In the virtual appliance console, log in as the root user.
2. Run the following command: $ /opt/vmware/hms/bin/hms-configtool -cmd upgrade -configfile /opt/vmware/hms/conf/hms-configuration.xml

   The DB schema upgrade starts.

3. Wait for the DB upgrade procedure to complete.
4. In the vSphere Replication VAMI, navigate to the Configuration tab, and complete the SSO registration of the appliance.