Vertica node crashed in DX NetOps Data Repository


Article ID: 48638


Updated On:


CA Performance Management - Usage and Administration DX NetOps


In a multi-node ksafe Vertica cluster one of the nodes has crashed. The exception in the vertica.log shows the following.

2013-08-19 08:09:05.086 Main:0x1ded8080 <PANIC> @v_repository_node0003: VX001/2973: Data consistency problems found; startup aborted
HINT: Check that all file systems are properly mounted. Also, the
--force option can be used to delete corrupted data and recover from the cluster
LOCATION: mainEntryPoint,
2013-08-19 08:09:05.142 Main:0x1ded8080 [Main] <PANIC> Wrote backtrace to


All supported DX NetOps Performance Management releases


In order to force the recovery of the database on the problem node, run the following command. Run it as the database user (default: dradmin) from the (default path) /opt/vertica/bin directory.

./admintools -t restart_node -d <DATABASE_NAME> -p <PASSWORD> -F -s <IP_Address_of_Node

Open adminTools and choose option #1 to "View Database Cluster State" the status should be "Recovering" This process may take a significant amount of time depending on the size of the database but once it is complete you will be able to successfully start the database on the node.

Note: you can run ./adminTools -h to get a full list of adminTools options.