Getting "PANIC: could not read from control file: Success" when trying to run a full DB vacuum on X-Series
Article ID: 168828
Either in Postgres log files in /cbconfig/conf/var/lib/pgsql/data/pg_log directory or on console DB vacuuming related messages similar to the following are seen:
FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres" HINT: Stop the postmaster and use a standalone backend to vacuum database "postgres". LOG: could not close temporary statistics file "global/pgstat.tmp": No space left on device
Note: "Postgres" DB is illustrative here, messages may concern other existing DBs as well.
Other symptoms include:
100% utilization of /cbconfig partition 100% utilization of /var partition
The following procedure was used to fully vacuum a DB:
# /etc/init.d/postgresql stop (stopping the DB, likely already not running) # su - postgres # echo "VACUUM FULL;" | postgres -D /var/lib/pgsql/data <DBname>
The last command produces the following output: PANIC: could not read from control file: Success Aborted
Checking file /cbconfig/conf/var/lib/pgsql/data/global/pg_control we see the file has zero size: -rw------- 1 postgres postgres 0 Oct 13 22:49 pg_control
On a healthy system, the file looks as follows: -rwx------ 1 postgres postgres 8192 Oct 16 13:17 pg_control
X-series Postgres database(s) running on CPM not vacuumed on time.
/cbconfig partition filled up completely with Postgres logs containing vacuum related messages
Potsgres service not able to run and create vital control file(s) due to lack of disk space
Although it is theoretically possible to recover pg_control file content so that Postgres would eventually start, to assure system consistency and to avoid running into other possible issues, a clean re-installation of XOS is recommended.
To prevent from such issues happening in future, it is necessary to make sure the second line shown below (marked in bold) is added to Postgres cron job file after system is re-installed: