vstdb doesn't start. FATAL: required WAL directory "pg_wal" does not exist

Products

• VMware vCenter Server

Issue/Introduction

The vstats service crashes when starting:

service-control --start vstats "An error occurred while starting service"

The postgresql log files will have the following message:

/var/log/vmware/vpostgres/postgresql.log:

yyyy-mm-dd HH:MM:ss.xxx UTC 646700f5.ad64 0   FATAL:  required WAL directory "pg_wal" does not exist

Also, VAMI Backups fail when attempted.

Cause

This occurs when the pg_wal directory is not present or the link has been deleted, the directory does not have access permissions, or wal files are not synchronized with Postgres vtsdb.

Resolution

Restore the vtsdb database checkpoint to start vtsdb and vstats services by performing the following:

1. Make sure that the /storage/vtsdblog/vtsdb/pg_wal directory exists, and link /storage/vtsdb/vtsdbdata/pg_wal exists and it has the correct permissions:
   # ls -l /storage/vtsdblog/vtsdb/
total 4
drwx------ 3 vtsdbuser root 4096 MM DD HH:MM pg_wal
# ls -l /storage/vtsdb/vtsdbdata/pg_wal
lrwxrwxrwx 1 vtsdbuser users 30 MM DD HH:MM /storage/vtsdb/vtsdbdata/pg_wal -> /storage/vtsdblog/vtsdb/pg_wal
2. If it doesn't exist, create it:
   
   # mkdir /storage/vtsdblog/vtsdb/pg_wal
# ln -s /storage/vtsdblog/vtsdb/pg_wal /storage/vtsdb/vtsdbdata/pg_wal
# chown -h vtsdbuser:users /storage/vtsdb/vtsdbdata/pg_wal
   
   
3. Check link permissions:
   
   # ls -l /storage/vtsdb/vtsdbdata/pg_wal
lrwxrwxrwx 1 vtsdbuser users 30 MM DD HH:MM /storage/vtsdb/vtsdbdata/pg_wal -> /storage/vtsdblog/vtsdb/pg_wal

4. the owner should be vtsdbuser:users, if not, change it:
   
   # chown -H vtsdbuser:users /storage/vtsdblog/vtsdb/pg_wal


5. Stop services:
   service-control --stop vtsdb
service-control --stop vstats


6. reset wal files:
   # su - vtsdbuser -s /bin/bash -c "/opt/vmware/vpostgres/current/bin/pg_resetwal -f /storage/vtsdb/vtsdbdata"


7. start services:
   # service-control --start vtsdb
# service-control --start vstats

NOTE: If it doesn't work, try stopping all services and starting them again.

Once this has been completed, try a VAMI based backup again.
 

Additional Information