Getting "PANIC: could not read from control file: Success" when trying to run a full DB vacuum on X-Series

book

Article ID: 168828

calendar_today

Updated On:

Products

CPM XOS

Issue/Introduction

Either in Postgres log files in /cbconfig/conf/var/lib/pgsql/data/pg_log directory or on console DB vacuuming related messages similar to the following are seen:

FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres" 
HINT: Stop the postmaster and use a standalone backend to vacuum database "postgres".
LOG:  could not close temporary statistics file "global/pgstat.tmp": No space left on device


Note: "Postgres" DB is illustrative here, messages may concern other existing DBs as well.

Other symptoms include:

100% utilization of
/cbconfig partition
100% utilization of 
/var partition

The following procedure was used to fully vacuum a DB:


# /etc/init.d/postgresql stop  (stopping the DB, likely already not running) 
# su - postgres 
# echo "VACUUM FULL;" | postgres -D /var/lib/pgsql/data <DBname>


The last command produces the following output:
PANIC: could not read from control file: Success 
Aborted 
 

Checking file /cbconfig/conf/var/lib/pgsql/data/global/pg_control we see the file has zero size:
-rw------- 1 postgres postgres 0 Oct 13 22:49 pg_control 

On a healthy system, the file looks as follows:
-rwx------ 1 postgres postgres 8192 Oct 16 13:17 pg_control

 

Cause

  1. X-series Postgres database(s) running on CPM not vacuumed on time.
  2. /cbconfig partition filled up completely with Postgres logs containing vacuum related messages
  3. Potsgres service not able to run and create vital control file(s) due to lack of disk space

Resolution

Although it is theoretically possible to recover pg_control file content so that Postgres would eventually start, to assure system consistency and to avoid running into other possible issues, a clean re-installation of XOS is recommended.

To prevent from such issues happening in future, it is necessary to make sure the second line shown below (marked in bold) is added to Postgres cron job file after system is re-installed:

0 2 * * 1-6 /usr/bin/vacuumdb -z -d running-config 
0 2 * * 0 /usr/bin/vacuumdb -z -a 
0 */2 * * * /usr/bin/vacuumdb -z -t webSessionTable -d running-config 
0 */2 * * * /usr/bin/vacuumdb -z -t alarmTable -d running-config

This will make sure all DBs are vacuumed every Sunday at 2:00am.

This cron extension was added to XOS 9.6.8, 9.7.2 and higher. 

 

Workaround

N/A