vPostgres service fails to start With PANIC: replication slot file "pg_replslot/vpg_archiver/state" has wrong magic number
search cancel

vPostgres service fails to start With PANIC: replication slot file "pg_replslot/vpg_archiver/state" has wrong magic number

book

Article ID: 368814

calendar_today

Updated On: 05-09-2025

Products

VMware vCenter Server

Issue/Introduction

  • vPostgres service fails to start with error "wrong magic number".
  • This typically occurs after a filesystem check failure or when recovering from emergency mode.
  • Log file /var/log/vmware/vmon/vmon.log will show similar entries to: 

YYYY-MM-DD-THH:MM:SS In(05) host-XXXX Received start request for vmware-vpostgres
YYYY-MM-DD-THH:MM:SS In(05) host-XXXX <vmware-vpostgres-prestart> Constructed command: /opt/vmware/vpostgres/current/scripts/pg_pre_start
YYYY-MM-DD-THH:MM:SS Wa(03) host-XXXX <vmware-vpostgres> Service pre-start command's stderr: Issuing signal KILL on all PostgreSQL processes owned by OS user vpostgres
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX Traceback (most recent call last):
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX   File "/opt/vmware/vpostgres/13/share/python-modules/vpostgres_cis/telemetry.py", line 247, in pushdata
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX     response = urllib.request.urlopen(request)
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX   File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX     return opener.open(url, data, timeout)
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX   File "/usr/lib/python3.7/urllib/request.py", line 531, in open
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX     response = meth(req, response)
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX   File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX     'http', request, response, code, msg, hdrs)
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX   File "/usr/lib/python3.7/urllib/request.py", line 569, in error
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX     return self._call_chain(*args)
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX   File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX     result = func(*args)
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX   File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX     raise HTTPError(req.full_url, code, msg, hdrs, fp)
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX urllib.error.HTTPError: HTTP Error 503: Service Unavailable
YYYY-MM-DD-THH:MM:SS Wa(03) host-XXXX <vmware-vpostgres> Service pre-start command's stderr: yyyy-mm-ddThh:mm:ss.153Z  Failed to push telemetry data for startup
YYYY-MM-DD-THH:MM:SS Wa(03)+ host-XXXX
YYYY-MM-DD-THH:MM:SS Wa(03) host-XXXX <vmware-vpostgres> Service pre-start command's stderr: Could not generate telemetry data
YYYY-MM-DD-THH:MM:SS In(05) host-XXXX <vmware-vpostgres> Service pre-start command completed successfully.
YYYY-MM-DD-THH:MM:SS In(05) host-XXXX <vmware-vpostgres> Constructed command: /opt/vmware/vpostgres/current/bin/postgres -D /storage/db/vpostgres
YYYY-MM-DD-THH:MM:SS host-XXXX Client info Uid=0,Gid=0,Pid=34887,Comm=(vmon-coredumper),PPid=2,Comm=(kthreadd),PPid=0
YYYY-MM-DD-THH:MM:SS Wa(03) host-XXXX [ReadSvcSubStartupData] No startup information from vmware-vpostgres.
YYYY-MM-DD-THH:MM:SS Wa(03) host-XXXX <vmware-vpostgres> Service exited. Exit code 1

  • Log file /var/log/vmware/vpostgres/postgresql-xx.log will show similar entries to: 

YYYY-MM-DD-THH:MM:SS 67a573d7.26bf 0   LOG:  listening on IPv6 address "::", port 5432
YYYY-MM-DD-THH:MM:SS 67a573d7.26bf 0   LOG:  listening on Unix socket "/var/run/vpostgres/.s.PGSQL.5432"
YYYY-MM-DD-THH:MM:SS 67a573d7.26c1 0   LOG:  database system was interrupted; last known up at 2024-09-20 03:07:24 UTC
YYYY-MM-DD-THH:MM:SS 67a573d7.26c1 0   PANIC:  replication slot file "pg_replslot/vpg_archiver/state" has wrong magic number: 875771188 instead of 17112225
YYYY-MM-DD-THH:MM:SS 67a573d7.26bf 0   LOG:  startup process (PID 9921) was terminated by signal 6: Aborted
YYYY-MM-DD-THH:MM:SS 67a573d7.26bf 0   LOG:  aborting startup due to startup process failure
YYYY-MM-DD-THH:MM:SS 67a573d7.26bf 0   LOG:  database system is shut down

Environment

vCenter Server 7.X
vCenter Server 8.X

 

Cause

This issue occurs due to the corrupt pg_replslot/vpg_archiver/state file. 

Resolution

Before proceeding with the steps below, take both backup and a snapshot of the vCenter Server Appliance. If the vCenter is part of a Enhanced Linked Mode (ELM) replication setup, also take a backup or offline (powered off) snapshot of all replicating vCenter ELM nodes.

  • SSH to vCenter
  • Navigate to /storage/db/vpostgres/pg_replslot
  • Remove file by issuing command,
    • rm -r vpg_archiver
  • Start vpostgres service using the command,
    • service-control --start vmware-vpostgres

Note: The state file gets re-created as we delete everything from directory "vpg_archiver" which is refreshed (recreated) when we issue the command to start the vpostgres service.

Additional Information