One machine from Cluster environment is going down

Products

• CA Infrastructure Management
• CA Performance Management - Usage and Administration

Issue/Introduction

One of the machines that has Vertica DB went down
Only one machine is going down.0.4 ENV: DEV/TEST ENT: YES KB & COMMUNITIES: HAVE NOT FOUND ANYTHING RELEVANT

Environment

Pm 3.2
VERTICA: 8.1.0.4

Cause

k-safe security syncronization
 

Resolution

If this a k-safe cluster and other nodes are UP, then please run below command on the down node. This command will start the 'node' in force mode, and it will remove all the corrupt data files and same will be recovered from other nodes in the cluster.

$admintools -t restart_node -F -s <this_Hostname_or_IP> -d <dbname>

Please replace hostname/IP and database name in the command.

Note1: This should to stop all nodes for synchronization and start using /opt/vertica/bin/adminTools so try to do this command out of business hours because it can take a long time to recover

Note2: Before executing this command run the Vertica Backup of node that is working fine

Additional Information

For standalone environment see  Document ID : KB000037369
Unable to start a single node Vertica database
https://comm.support.ca.com/kb/unable-to-start-a-single-node-vertica-database/kb000037369