VMware Identity Manager (vIDM) node will not come up, with vpostgres showing error 'pg_ctl: directory "/db/data" is not a database cluster directory'

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Checking status of postgres or trying to (re)start it leads to an error about the /db/data directory

         /etc/init.d/vpostgres status
         /etc/init.d/vpostgres restart

       pg_ctl: directory "/db/data" is not a database cluster directory

• Aria Lifecycle Cluster-Auto-recovery doesn't fix the cluster

Environment

VMware Identity Manager 3.3.7

Cause

There can be numerous causes. This article provides two possible issues and we can search for both:

1. The (internal) passwords for psql "repl" or "postgres" users may be out of sync with stored values
2. There may be non-standard files in the /db/data directory which psql is not happy about. The /db/data folder can be empty or only contain the recovery.conf file on the secondary vIDM nodes if the 'repl' user password stored on the master DB is not matching.

Resolution

Before attempting any of this, please take non-memory snapshots of all Identity Manager nodes

These two issues can be checked for in either order:

Issue 1: Passwords

If the password for psql users 'repl' or 'postgres' do not match these files, then it will need to be updated:

• Check /db/data/recovery_1st_stage  for user `repl`:
  1. grep 'password=' /db/data/recovery_1st_stage
  2. Try to log in with that password:
     PGPASSWORD=<Password from the above command> psql -U repl -h master -d postgres
• /usr/local/etc/pgpool.conf for user `postgres`
  1. grep 'recovery_password =' /usr/local/etc/pgpool.conf
  2. Try to log in with that password:
     PGPASSWORD=<Password from the above command>  psql -U postgres -h master -d postgres

If either login is not successful then that password needs to be updated in the PSQL prompt on the master DB, with this query:

        Connect Postgres DB using the command psql -U postgres saas 

ALTER USER repl PASSWORD '<new password>'

or

ALTER USER postgres PASSWORD '<new password>'

 

Issue 2: Non-standard files

The /db/data directory is not a place to leave files manually. It is for the postreSQL DB data only.

1. Snapshot all 3 nodes - non-memory without suspending
2. Search for any such files:
   ls -Al /db/data
3. Typical culprits may have ownership other than postgres, such as owned by root
4. If you have one working node, you could also run ls -Al /db/data there and compare the listings
5. Compare also the directory listings in ls -Al /db/data/pg_xlog
6. Any standard psql files owned by root somegroup should have their owner changed:
   • chown postgres:users pg_standard_file
     
     or, with great caution, move or remove and file which doesn't belong there:
   • mv /db/data/strangefile.txt /root/