Smarts NCM: JBOSS service not starting; Errors in server.postmaster and JBOSS.log files: invalid page header

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:


Smarts NCM application not starting as expected

The following error is showing in the JBOSS.log which will prevent JBOSS from started properly (timing out) and will not allow the NCM GUI to open.  

JBoss server did not start because this error:
Deployment "vc:service=VoyenceControlConfigStartup" is in error due to the following reason(s): org.postgresql.util.PSQLException: ERROR: invalid page header in block 3 of relation base/16385/16551, **ERROR**


In the $VOYENCE_HOME/db/controldb/logs/server.postmaster, invalid page header error:
SessionId:53cecfc2.3647 - TransId:0 - ERROR:  invalid page header in block 3 of relation base/16385/16551

SessionId:53cecfc2.3647 - TransId:0 - STATEMENT:  select configitem0_.config_item_id as config1_106_, configitem0_.version as version106_, configitem0_.config_type as config3_106_, configitem0_.name as name106_, configitem0_.value as value106_, configitem0_.editable as editable106_, configitem0_.default_value as default7_106_ from cm_config_item configitem0_

Environment

VMware Smart Assurance - NCM

Cause

This is an issue with a corrupt table in the NCM database.

The error in the $VOYENCE_HOME/db/controldb/logs/server.postmaster log will have the details of the table name.

For instance, in the example below, the table name is: cm_config_item

SessionId:53cecfc2.3647 - TransId:0 - ERROR:  invalid page header in block 3 of relation base/16385/16551

SessionId:53cecfc2.3647 - TransId:0 - STATEMENT:  select configitem0_.config_item_id as config1_106_, configitem0_.version as version106_, configitem0_.config_type as config3_106_, configitem0_.name as name106_, configitem0_.value as value106_, configitem0_.editable as editable106_, configitem0_.default_value as default7_106_ from cm_config_item configitem0_

Resolution

This is an issue with the database table mentioned in the $VOYENCE_HOME/db/controldb/logs/server.postmaster log.

In this example:
cm_config_item
  
Run the following based on the table above:
The following actions below must be performed with all the services on the Application Server, Device Server and Report Server are stopped and the controldb is the only thing running


Log into the Database by doing the following at the command prompt:
su - pgdba
psql voyencedb voyence
(It will ask for the Database password) 
Enter DB password.

Once you are logged in to DB, execute below commands:
1. SET zero_damaged_pages = on;
2. VACUUM FULL damaged_table;
3. REINDEX TABLE damaged_table; 



For example: In my case I am having an issue with table cm_config_item so I will execute the following commands to fix this issue:

1)
database=# SET zero_damaged_pages = on;
SET
database=# VACUUM FULL cm_config_item;
WARNING: invalid page header in block xxx of relation base/yyy/zzz; zeroing out page
[..]
REINDEX TABLE cm_config_item;

This will remove the affected items and add 0 to the affected cells.  

2) Verify that the table can be read now by executing a simple sql command to open the contents in that table:
select * FROM damaged_table;

Example:
select * FROM cm_config_item;

This should populate the cell information inside cm_config_item table.  

3) Restart vcmaster after performing these steps. This should restart JBOSS properly without any error messages regarding invalid page header. 

Launch NCM and test by logging in and performing a job. 

Additional Information

Please see following KB for details on how to run queries aganist the database:

https://emc--c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004Jyi