Vertica Cluster stays in 'Initializing/Down' state, Performance Management down

Products

• CA Infrastructure Management
• CA Performance Management
• Network Observability

Issue/Introduction

After patching the OS the Data Repository cluster nodes run on the server required a reboot.

After the reboot the Vertica database on the Data Repository cluster fails to start.

The database was shut down properly before OS patching and reboot.

Vertica database stays in "Initializing/Down" state after reboot.

Vertica database fails to start showing constant "Initializing/Down" state after reboot.

Environment

Release : 3.7

Component : IM Reporting / Admin / Configuration

Cause

Undetermined. Errors observed in logging are generic in nature. They indicate general communication failures for some nodes while they wait for other nodes to complete start up to advertise themselves to the cluster as ready/available.

Resolution

Note: Without validation of a proper cause and solution, this method may result in data loss. If this is a production critical data base, open a support case for further assistance.

Two things are seen to resolve this. First try restarting Vertica on all hosts by shutting it down on each host, then starting the database.

1. Log in as dradmin or equivalent DR DB install owner/administrator user.
2. Launch adminTools from /opt/vertica/bin.
3. Select option 7 "Advanced Menu"
4. Select option 2 "Stop Vertica on Host"
   1. Choose all nodes and try to stop them.
   2. If any fail to stop, try repeating the process on that node itself, launching adminTools on that node itself.
5. Once all nodes are shut down, go to "Main Menu" in adminTools.
6. Choose option 3 "Start Database"

If that fails to get the database started, try rebooting the nodes to see if the database starts after a clean reboot.