Vertica node crashed in CAPM Data Repository

Products

• CA Infrastructure Performance

Issue/Introduction

Description:

In a multi node ksafe Vertica cluster one of the nodes has crashed. The exception in the vertica.log shows ....

2013-08-19 08:09:05.086 Main:0x1ded8080 <PANIC> @v_repository_node0003: VX001/2973: Data consistency problems found; startup aborted
HINT: Check that all file systems are properly mounted. Also, the
--force option can be used to delete corrupted data and recover from thecluster

LOCATION: mainEntryPoint,
/scratch_a/release/vbuild/vertica/Basics/vertica.cpp:1055

2013-08-19 08:09:05.142 Main:0x1ded8080 [Main] <PANIC> Wrote backtrace to ErrorReport.txt

 

Environment

CAPM 3.x

Resolution

In order to force the recovery of the database on the problem node, run the following command as the database user (default: dradmin):

./adminTools -t restart_node -d <DATABASE_NAME> -p <PASSWORD> -F -s <IP_Address_of_Node> 

 

Now, when you open adminTools and choose option #1 to "View Database Cluster State" the status should be "Recovering" This process may take a significant amount of time depending on the size of the database but once it is complete you will be able to successfully start the database on the node.

Note: you can run ./adminTools -h to get a full list of adminTools options.