vCenter services fails to start due to Incorrect vPostgres configuration files

Products

• VMware vCenter Server 8.0

Issue/Introduction

• vCenter Server services (sps, vpxd, vmware-postgres-archiver, vpxd-svcs, vc-ws1a-broker) fails to start and the vCenter UI reports "503 Service Unavailable" error.
• The issue often occurs after a failed vCenter upgrade or unsuccessful snapshot rollback operation.
• When attempting to start services, the following error is observed in the console:
  
  Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, vpxd, vmware-postgres-archiver, vpxd-svcs, vc-ws1a-broker services. Error: Operation timed out


• /var/log/vmware/vpxd-svcs/vpxd-svcs.log contains errors indicating SCRAM authentication failures:
  
  Caused by: com.vmware.cis.core.kv.impl.Exception.KVExecutionException: SQL Error:
Caused by: org.postgresql.util.PSQLException: The server requested SCRAM-based authentication, but no password was provided.


• /var/log/vmware/vpxd/vpxd.log contains the following socket connection failures:
  
  error vpxd[72949] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Failed to connect to database: ODBC error: (08001) - [unixODBC]connection to server on socket "/var/run/vpostgres/.s.PGSQL.5432" failed: fe_sendauth: no password supplied

Environment

VMware vCenter Server 8.x

Cause

The issue is caused by incorrect or corrupted entries within the pg_hba.conf and pg_ident.conf configuration files located in the /storage/db/vpostgres directory, which prevents vCenter services from successfully authenticating to the local vPostgres database.

Resolution

1. Identify a working vCenter Server appliance running the exact same version and build as the affected system.
2. Copy the pg_hba.conf and pg_ident.conf files from the /storage/db/vpostgres/ directory of the working vCenter Server.
3. Transfer the copied files to the affected vCenter Server, overwriting the existing files in the /storage/db/vpostgres/ directory.
4. Set the correct ownership on the newly replaced files by executing the command:
   chown vpostgres:vpgmongrp /storage/db/vpostgres/pg_hba.conf /storage/db/vpostgres/pg_ident.conf
   
   
5. Restart the vCenter Server services to apply the changes and restore functionality.
   
   service-control --stop --all
service-control --start --all

Additional Information

Stopping, starting or restarting VMware vCenter Server Appliance services