Vertica Node Down That Is Part Of A Three Node Vertica Cluster

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

The Vertica node had crashed due to a hardware failure.  What is the process for recovering a failed Vertica node that is part of a three node Vertica cluster.

Environment

DX NetOps Performance Management any version

Cause

Hardware failure.

Resolution

Vertica provided the following steps and these are official:

Vertica recommended the following steps to recover the failed node:

1. Added a new node
$ sudo /opt/vertica/sbin/update_vertica --add-hosts <xx.xx.xx.xx> --rpm verticaxxxxxx.rpm -u dbadmin

2. Placed new node in STANDBY:
ALTER NODE <node-name> STANDBY;

3. Manually Replacing a Node, enter the command:
ALTER NODE <node-name> REPLACE WITH <standby-node-name>;

4. Remove the node from the database using drop command.
DROP NODE <node-name> CASCADE;

5. Remove any occurrence of the down node in the admintools.conf file (/opt/vertica/config/admintools.conf, manually edit it).

6. Distribute the edited admintools.conf in all the nodes in the cluster by using admintools. Command:
$ admintools -t distribute_config_files

7. Perform the query below to confirm the node is gone.
SELECT * FROM nodes;

Additional Information

The customer performed the following steps instead (not from Vertica):

1. Installed Vertica on the lost node as single instance (OS was setup from scratch).

2. I started the Vertica database from the 1st node and the 3rd node was recovering

Now the cluster is back to three nodes.