vmware-postgres-archiver service does not start after reboot or after taking offline snapshot

Products

• VMware vCenter Server

Issue/Introduction

• Service vmware-postgres-archiver fails to start. 
• In /var/log/vmware/cloudvm/service-control.log, see following error: 
  
  service-control.log
  
  YYYY-MM-DDTHH:MM:SS.###<Time_zone> ERROR service-control Error executing start on service vmware-postgres-archiver. Details {
      "detail": [
          {
              "id": "install.ciscommon.service.failstart",
              "translatable": "An error occurred while starting service '%(0)s'",
              "args": [
                  "vmware-postgres-archiver"
              ],
              "localized": "An error occurred while starting service 'vmware-postgres-archiver'"
          }
      ],
      "componentKey": null,
      "problemId": null,
      "resolution": null
  }
  YYYY-MM-DDTHH:MM:SS.###<Time_zone> ERROR service-control Service-control failed. Error: {
      "detail": [
          {
              "id": "install.ciscommon.service.failstart",
              "translatable": "An error occurred while starting service '%(0)s'",
              "args": [
                  "vmware-postgres-archiver"
              ],
              "localized": "An error occurred while starting service 'vmware-postgres-archiver'"
          }
      ],
      "componentKey": null,
      "problemId": null,
      "resolution": null
  
  
• An alarm will be triggered on the vSphere UI: 'PostgreSQL Archiver Service Alarm (CRITICAL)'

• In /var/log/vmware/vmon/vmon.log, see error similar to:

  YYYY-MM-DDTHH:MM:SSSZ Wa(03) host-2455 <vmware-postgres-archiver> Service pre-start command's stderr: Failed to send the telemetry data due to Service Unavailable
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455 Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455   File "/opt/vmware/vpostgres/14/share/python-modules/vpostgres_cis/telemetry.py", line 249, in pushdata
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455     response = urllib.request.urlopen(request, timeout=MAX_TIMEOUT)
........
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455   File "/usr/lib/python3.10/urllib/request.py", line 563, in error
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455     return self._call_chain(*args)
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455   File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455     result = func(*args)
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455   File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455     raise HTTPError(req.full_url, code, msg, hdrs, fp)
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455 urllib.error.HTTPError: HTTP Error 503: Service Unavailable
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455 YYYY-MM-DDTHH:MM:SSSZ  Failed to push telemetry data for archiver
YYYY-MM-DDTHH:MM:SSSZ Wa(03)+ host-2455 
YYYY-MM-DDTHH:MM:SSSZ In(05) host-2455 <vmware-postgres-archiver> Service pre-start command completed successfully.
YYYY-MM-DDTHH:MM:SSSZ In(05) host-2455 <vmware-postgres-archiver> Constructed command: /opt/vmware/vpostgres/current/bin/pg_archiver --directory /storage/archive/vpostgres --compress 1 --slot vpg_archiver --health-naptime 60 -n --verbose --purge -d service=archiver --health-expiration 90 --health-file /dev/shm/vmware-postgres-archiver-health-status.xml 
YYYY-MM-DDTHH:MM:SSSZ Wa(03) host-2455 <vmware-postgres-archiver> Service exited. Exit code 1

• In /var/log/vmware/vpostgres/pg_archiver.log.stderr, see error similar to:

  YYYY-MM-DDTHH:MM:SSSZ ERROR  pg_archiver could not open directory "/storage/archive/vpostgres": Permission denied
Starting service process with pid: 3252.
YYYY-MM-DDTHH:MM:SSSZ ERROR  pg_archiver could not open directory "/storage/archive/vpostgres": Permission denied
Starting service process with pid: 49752.

• On validating permission on the folder: /storage/archive/vpostgres looks like:

  [ /storage/archive ]# ls -ltrah
  
  drwx------   2 \

Environment

vCenter Server 8.x

Cause

This issue occurs due to incorrect permissions set on the folder: /storage/archive/vpostgres

Resolution

• Change ownership for the folder: /storage/archive/vpostgres using the below command:

chown -R vpostgres:vpgmongrp vpostgres/

• Updated permission on the folder: /storage/archive/vpostgres will appear as below:

[ /storage/archive ]# ls -ltrah

drwx------   2 vpostgres vpgmongrp 696K Jul  4 08:52 vpostgres

• Start vmware-postgres-archiver service using the below command:

vmon-cli --start vmware-postgres-archiver