vmware-postgres-archiver service fails to start

Products

• VMware vCenter Server

Issue/Introduction

- Service vmware-postgres-archiver fails to start

- vmware-postgres-archiver is restarting itself continuously

- VCSA file-based backups are failing with error: "All required services are not up! Stopped services: 'vmware-postgres-archiver'."

- /storage/archive has gone into read-only in the past

Environment

vCenter Server 6.x

vCenter Server 7.x

vCenter Server 8.x

Cause

1) The vpostgres replication slot vpg_archiver was removed because it was lagging too much behind. This is expected. Note that this forces service vmware-postgres-archiver to stop.
2) The service is restarted automatically by vMon, and continuously fails to start because it has lost track of the WAL files
4) Because of the failure in 3) the archiver has finished in a state where it recreated a new replication slot, but it could not remove the past WAL segments so it kept restarting and willing to replicate from a WAL position which was already removed in the server.
5) The manual steps enforce the cleanup to happen, which was not happening in step 3), and failed in step 4) because of a new replication slot re-created in the middle.

 

"ERROR  pg_archiver unexpected termination of replication stream: ERROR:  requested WAL segment 0000000100000218000000D2 has already been removed"

Resolution

Take a snapshot/backup of the vCenter, referencing the following KB if using ELM;
https://knowledge.broadcom.com/external/article/313886/vmware-vcenter-in-enhanced-linked-mode-p.html

1) Stop the archiver service:
vmon-cli --stop vmware-postgres-archiver
2) Drop the replication slot that the archiver uses:
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT pg_drop_replication_slot('vpg_archiver');" postgres
3) Restart the archiver, which will automatically clean up past archives and create a new replication slot:
vmon-cli --start vmware-postgres-archiver.