vpxd service on vCenter will not start due to corrupt vPostgres transaction logs

Products

• VMware vCenter Server

Issue/Introduction

When attempting to start the vpxd service, the following error appears in the vpxd.log:

Log file: /var/log/vmware/vpxd/vpxd.log

Failed to connect to database: ODBC error: (08001) - [unixODBC] Could not connect to the server; --> Connection refused [XXX.X.X.XX:XXXX]

Additionally, the PostgreSQL logs indicate the following error 

Log file: /var/log/vmware/vpostgres/postgresql.log

LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record.

Environment

VMware vCenter Server

Cause

vCenter Server service is unable to start as due to database corruption or missing checkpoint records in PostgreSQL. 

Postgres transaction logs can become corrupted when vCenter is abruptly shutdown, disk becomes full, or due to hardware failures.

Resolution

To resolve the issue, the Postgres transaction logs need to be reset. 

Warning:

• Resetting the transaction logs can cause data loss.
• vCenter Server may need to be restored from backup if resetting the transaction logs fails or if corruption persists.

Note: Shutdown and power off the vCenter Server appliance. Take a snapshot of the VM. This snapshot can be used to roll back any changes.

1. Power on the vCenter Server.
2. Stop all services using the below command
   

   service-control --stop --all

    

3. Switch to vpostgres user.
   

   su vpostgres -s /bin/sh 

4. Reset the transaction log.
   
   For vCenter Server Appliance 6.5.x and 6.7.x:
   

   /opt/vmware/vpostgres/current/bin/pg_resetxlog -f  /storage/db/vpostgres

   
   For vCenter Server 7.0 and later:
   

   /opt/vmware/vpostgres/current/bin/pg_resetwal -f /storage/db/vpostgres

   
   
   
5. Exit vpostgres user session
   

   exit

6. Start all services.
   

   service-control --start --all

7. After vCenter Server starts successfully. Review vCenter functionality and delete the snapshot.

Additional Information