vFabric Postgres fails with the error: invalid page header in block
search cancel

vFabric Postgres fails with the error: invalid page header in block

book

Article ID: 341471

calendar_today

Updated On:

Products

VMware

Issue/Introduction

This article provides steps on zeroing out and rebuilding damaged Postgres database pages.

Symptoms:
  • The Postgres database does not start
  • pg_dump backups fail
  • The Postgres application may fail when accessing data
  • In the Postgres logs, you see errors similar to:

    2012-11-14 23:00:43.451 UTC alivevm alive ERROR: invalid page header in block 57300 of relation pg_tblspc/16385/PG_9.0_201106101/16386/16873
    2012-11-14 23:00:43.451 UTC alivevm alive STATEMENT: select r.RESOURCE_ID, r.ATTRKEY_ID, r.RKATTRIB_ID from ResourceAttributeKey r where (1=1) order by 1


Environment

VMware vFabric Postgres Standard Edition 9.1.x
VMware vFabric Postgres Standard Edition 9

Cause

This issue may occur due to these situations:
  • The underlying storage has failed unexpectedly.
  • The application may have been writing to the database when storage went offline.
  • Disk/hardware corruption.
  • Another application or process overwrote data on the same disk files.

Resolution

To resolve this issue:

  1. Log into the system that hosts the affected database with the Postgres super user or change to the Postgres user. For example:

    su postgres

  2. Back up the database files using a command similar to:

    cp -r /path/data/* /path/databu

  3. Start the psql utility by running a command similar to:

    psql -d dbname

  4. Set the parameter to zero pages automatically:

    dbname=# SET zero_damaged_pages = on;

  5. Force a check on the pages using a command similar to:

    dbname=# vacuum full tablename;

    Notes:
    • Replace tablename with the name retrieved from the output or error log. In the example log provided in this article, the table name is ResourceAttributeKey.
    • If the vacuum command fails with an error similar to:

      PANIC: checksum mismatch: disk has 0, should be 0x5f86405e filename pg_tblspc/16385/PG_9.0_201106101/16386/16873, BlockNum 57296, block specifier 16385/16386/16873/0/57296

      See Resolving checksum mismatch errors on VMware vPostgres Standard Edition 9.0/9.1.x (2030160).

  6. Repeat the process until the vacuum command succeeds without errors.
  7. Check the table by running a statement similar to:

    dbname=# SELECT count(*) FROM tablename;

    This is successful if the statement returns a number of rows without errors.


Additional Information

Resolving checksum mismatch errors in VMware vFabric Postgres (vPostgres) 9.0 / 9.1.x

Impact/Risks:
  • This procedure zeroes out pages and there will be loss of data in the affected table/indexes.
  • The database should start up after following this procedure, but it may not start if the rows lost are needed for application initialization. It may be necessary to rebuild rows according to the needs of the application.