vmware-postgres-archiver service crashes and fails to start

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vSphere Client  is not accessible with error :  "no healthy upstream".
  
  
• Restarting vmware-postgres-archiver fails:- # service-control --start vmware-postgres-archiver
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmware-postgres-archiver...
Error executing start on service vmware-postgres-archiver. Details {
    "resolution": null,
    "componentKey": null,
    "problemId": null,
    "detail": [
        {
            "args": [
                "vmware-postgres-archiver"
            ],
            "translatable": "An error occurred while starting service '%(0)s'",
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-postgres-archiver'"
        }
    ]
}
Service-control failed. Error: {
    "resolution": null,
    "componentKey": null,
    "problemId": null,
    "detail": [
        {
            "args": [
                "vmware-postgres-archiver"
            ],
            "translatable": "An error occurred while starting service '%(0)s'",
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-postgres-archiver'"
        }
    ]
}
  
  
• /var/log/vmware/vpostgres/pg_archiver.log.stderr 
  

  YYYY-MM-DDTH:M:S DEBUG  pg_archiver Updated startup LSN using segment file "000000010000014F00000006.gz.partial"
YYYY-MM-DDTH:M:S DEBUG  pg_archiver starting log streaming at 14F/6000000 (timeline 1)
YYYY-MM-DDTH:M:S ERROR  pg_archiver unexpected termination of replication stream: ERROR:  requested WAL segment 000000010000014F00000006 has already been removed
YYYY-MM-DDTH:M:S ERROR  pg_archiver disconnected

• You might not be able to backup the vCenter server using VAMi with below error:
   "Invalid vCenter Server Status: All required services are not up! Stopped services: 'vmware-postgres-archiver'." 

 

Environment

vCenter Server 7.x,

vCenter Server 8.x

Cause

Sudden power loss or Power outage could lead VCDB going down.

Issue can occur while performing a restart of services or while performing patching as well. 

This issue can also happen if you manually delete recent wal segments from /storage/archive partition.

Replication slot in pg database has the details of these WAL segments ,hence it reports requested WAL segment is already removed error.

Resolution

1. Take a vCenter snapshot

2. Log in to the vCenter Server Appliance through SSH.

3. Run this command to enable access the Bash shell:
       shell.set --enabled true

4. Type shell and press Enter.
   
   
5. Drop the replication slot that the archiver uses:
        /opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT pg_drop_replication_slot('vpg_archiver');" postgres
   
   
6. Restart the archiver, which will automatically clean up past archives and create a new replication slot:
        service-control --start vmware-postgres-archiver

Additional Information

To deal with /storage/archive partition utilization, please go through below kb article:

https://knowledge.broadcom.com/external/article/318803

You can exclude the /storage/archive utilization alerts on vROPS using below  KB:

https://knowledge.broadcom.com/external/article/315974/continuous-disk-space-alerts-for-storage.html