vPostgres service is failing to start due to errors in configuration files

Products

• VMware vCenter Server

Issue/Introduction

When attempting to start the vPostgres service on vCenter Server, you observe one or more of the following symptoms:

• Running `service-control --start vmware-vpostgres` returns an error: `An error occurred while starting service 'vmware-vpostgres'`.
• In `/var/log/vmware/vpostgres/serverlog.stderr`, you see syntax errors such as:
  
  • `LOG: syntax error in file "/storage/db/vpostgres/postgresql.conf.auto" line 1, near token ""` 
  • `FATAL: configuration file "/storage/db/vpostgres/postgresql.conf" contains errors` 
•  In `/var/log/vmware/vmon/vmon.log`, you see errors regarding health XML files:
  
  • `Error: Failed to read health xml file: /dev/shm/vmware-postgres-health-status.xml` 
  • `Error: [Errno 2] No such file or directory: '/dev/shm/vmware-postgres-health-status.xml'`

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter Server 9.0

Cause

This issue typically occurs due to one of the following:

• Corrupted `postgresql.conf.auto`, `postgresql.conf`, or `health_status_worker.conf` files in the `/storage/db/vpostgres/` directory, often following a disk space exhaustion event on the `/storage/db` partition.
• Missing or truncated health status XML files in the shared memory partition (`/dev/shm`) required by the `vmon` process to validate service health during initialization.

Resolution

To resolve the issue, follow the below steps:

1. Log in to the vCenter Server Appliance as root via SSH.
2. Check the errors in the /var/log/vmware/vpostgres/serverlog.stderr.
   
   
3. Note the configuration files that contains errors.
   • LOG:  syntax error in file "/storage/db/vpostgres/postgresql.conf.auto" line 3, near token ""
   • LOG:  syntax error in file "/storage/db/vpostgres/health_status_worker.conf" line 1, near token ""
     
     
4. Open files in vi and compare the content from other working vCenter Server running on the same build version.
   • A typical postgresql.conf.auto file example:

     # /storage/db/vpostgres/postgresql.conf.auto
     # File automatically generated at vPostgres firstboot
     # Do not update manually
     maintenance_work_mem = '201 MB'
     work_mem = '420 kB'
     temp_buffers = '840 kB'
     shared_buffers = '1008 MB'
     max_wal_size = '17512 MB'
     min_wal_size = '17512 MB'
     max_connections = 737

   • A typical health_status_worker.conf file example:

     health_status_worker.naptime = 30
     health_status_worker.expiration_time = 90
     health_status_worker.log = true
     health_status_worker.status_file_path = '/dev/shm/vmware-postgres-health-status.xml'
     health_status_worker.memory_allowed = '2018 MB'
     health_status_worker.disk_space_threshold = 0.05
     health_status_worker.disk_space_log_threshold = 0.05
     health_status_worker.disk_space_wal_threshold = 0.2
     health_status_worker.status_file_group = 'vpgmongrp'
     health_status_worker.memory_file_path = '/sys/fs/cgroup/memory/system.slice/vmware-vpostgres.service/memory.stat'
     health_status_worker.memory_flag = 'rss'
     health_status_worker.replication_slot_check = true
     health_status_worker.replication_slot_drop_threshold = 0.9
     health_status_worker.replication_slot_force_checkpoint = on

   • The postgresql.conf file is vcenter specific so should be referenced but not copied directly.
     
     
5. Take a backups of these files.
   • #cp /storage/db/vpostgres/postgresql.conf.auto /storage/db/vpostgres/postgresql.conf.auto.bak
   • #cp /storage/db/vpostgres/health_status_worker.conf /storage/db/vpostgres/health_status_worker.conf.bak
     
     
6. Correct the files manually to match a known working configuration.  Update permissions as needed.

7. Verify the existence and content of the health status files in shared memory:

   • Run `ls -l /dev/shm/vmware-postgres*`.

   • If `vmware-postgres-health-status.xml` or `vmware-postgres-archiver-health-status.xml` are missing or have a file size of 0, they must be restored.

     • Restore missing health XML files:

       • If you have a high-availability (HA) pair or another vCenter node of the same version, copy the healthy `.xml` files from that node's `/dev/shm/` directory to the affected node.

       • If no other node is available, contact support for template files.
         
         

8. Start vPostgres service by running service-control --start vmware-vpostgres

Additional Information

Note:
The contents of the postgresql.conf.auto file will be auto updated with new values when vpostgres starts.