vPostgres Service Fails to Start in vCenter 8.0 U2

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

In a vCenter Server 8.0 U2 environment, you may observe the following symptoms:

• When accessing the vSphere Client in a browser, “no healthy upstream” is displayed.

• When checking services with service-control --status, vmware-vpostgres remains in start-pending.

• The following messages are recorded in vmon.log:

  • Service batch op START failed. Failed services: 'vmware-vpostgres'

  • <vmware-vpostgres> Service api-health command's stderr: Error getting service health. Error: Failed to read health xml file: /dev/shm/vmware-postgres-health-status.xml. Error: [Errno 2] No such file or directory: '/dev/shm/vmware-postgres-health-status.xml'

Environment

VMware vCenter Server 8.0

Cause

The database configuration file postgresql.conf is corrupted or was unintentionally modified.
The file may become extremely small and required startup parameters can be missing.

As a result, the pre-start process and API health check cannot complete successfully.

Resolution

Replace the corrupted postgresql.conf with a known-good default file from a healthy vCenter Server running the same version.

1. Log in to the vCenter Server via SSH as root.

2. Start the shell:

    shell

3. Back up the existing configuration file:

    cp /storage/db/vpostgres/postgresql.conf /storage/db/vpostgres/postgresql.conf.org

4. Copy a valid postgresql.conf obtained from a healthy vCenter Server (same version) to /storage/db/vpostgres/postgresql.conf.

   1. You can also obtain a sample file under /opt/vmware/vpostgres
      1. Firstly, you need to determine what version of postgres you are currently running, by running the below command
         
         /opt/vmware/vpostgres/current/bin/postgres --version

/opt/vmware/vpostgres/current/bin/postgres --version
         postgres (PostgreSQL) 14.19 (VMware Postgres 14.19.0-24974826 release)
         
         
      2. If for example you are like the above, running version 14, the sample file can be located at /usr/pgsql/14/share/postgresql/postgresql.conf.sample
      3. Copy this file to /storage/db/vpostgres/postgresql.conf
         
         cp -p /usr/pgsql/14/share/postgresql/postgresql.conf.sample /storage/db/vpostgres/postgresql.conf
         
         
5. Make sure the following include lines are present at the very bottom of /storage/db/vpostgres/postgresql.conf:

#vCenter Extesions & Includes

shared_preload_libraries = 'health_status_worker,pg_stat_statements'
session_preload_libraries = 'auto_explain'
max_locks_per_transaction = 128

ssl_ciphers = 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:AES256-SHA:AES128-SHA'
ssl_ecdh_curve = 'prime256v1'
ssl_min_protocol_version = 'TLSv1.2'
ssl_max_protocol_version = 'TLSv1.3'

include = 'postgresql.conf.auto'
include_if_exists = 'health_status_worker.conf'
include_if_exists = 'postgresql.conf.repl'
include_if_exists = 'standby.conf'
include_if_exists = 'stig.conf'

     6. Verify ownership and permissions:

• • Owner: vpostgres:vpgmongrp
    
    chown vpostgres:vpgmongrp /storage/db/vpostgres/postgresql.conf

  • Permissions: 600 (-rw-------)
    
    chmod 600 /storage/db/vpostgres/postgresql.conf

     7. Restart the vmware-vpostgres service

          service-control --restart vmware-vpostgres

    8. Restart all services

 service-control --stop --all
 service-control --start --all

You may need to address some issues for some of the other services. For example, pg_archiver requires the correct path to be specified in the newly created/pasted postgresql.conf file. It defaults to:

     unix_socket_directories = '/tmp'

     You need to change this to:

     unix_socket_directories = '/var/run/vpostgres'

     9. Once you do this, re-start the vmware-vpostgres service:

     service-control --restart vmware-vpostgres

     10. Ensure that the .s.PGSQL.5432 file is created in /var/run/vpostgres/, by running ls -la /var/run/vpostgres/

     11.  Restart all services

 service-control --stop --all
 service-control --start --all

         Wait approximately 15 minutes for services to start successfully.

   11. Your vcsa should now be available to log in to.

Additional Information

• If a “Permission denied” message is recorded due to insufficient write permissions to: /var/log/vmware/cloudvm/cloudvm-ram-size.log refer to the related KB: 
  Failed to start vCenter services due Permission issue in /var/log/vmware/cloudvm/cloudvm-ram-size.log