Either in Postgres log files in /cbconfig/conf/var/lib/pgsql/data/pg_log directory or on console DB vacuuming related messages similar to the following are seen:
FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres"
HINT: Stop the postmaster and use a standalone backend to vacuum database "postgres".
LOG: could not close temporary statistics file "global/pgstat.tmp": No space left on device
Note: "Postgres" DB is illustrative here, messages may concern other existing DBs as well.
Other symptoms include:
100% utilization of /cbconfig partition
100% utilization of /var partition
The following procedure was used to fully vacuum a DB:
# /etc/init.d/postgresql stop (stopping the DB, likely already not running)
# su - postgres
# echo "VACUUM FULL;" | postgres -D /var/lib/pgsql/data <DBname>
The last command produces the following output:
PANIC: could not read from control file: Success
Checking file /cbconfig/conf/var/lib/pgsql/data/global/pg_control we see the file has zero size:
-rw------- 1 postgres postgres 0 Oct 13 22:49 pg_control
On a healthy system, the file looks as follows:
-rwx------ 1 postgres postgres 8192 Oct 16 13:17 pg_control
Although it is theoretically possible to recover pg_control file content so that Postgres would eventually start, to assure system consistency and to avoid running into other possible issues, a clean re-installation of XOS is recommended.
To prevent from such issues happening in future, it is necessary to make sure the second line shown below (marked in bold) is added to Postgres cron job file after system is re-installed:
0 2 * * 1-6 /usr/bin/vacuumdb -z -d running-config
0 2 * * 0 /usr/bin/vacuumdb -z -a
0 */2 * * * /usr/bin/vacuumdb -z -t webSessionTable -d running-config
0 */2 * * * /usr/bin/vacuumdb -z -t alarmTable -d running-config
This will make sure all DBs are vacuumed every Sunday at 2:00am.
This cron extension was added to XOS 9.6.8, 9.7.2 and higher.