How to recover the Data Repository (Vertica) Database startup failed on one of the cluster node

Products

• CA Infrastructure Management
• CA Performance Management - Usage and Administration

Issue/Introduction

When starting Data Repository database, the following error appeared in dbLog.

*** Starting database: drdata *** 

Starting nodes: 

v_drdata_node0001 

v_drdata_node0002 

v_drdata_node0003 

Error: the vertica process for the database is running on the following hosts: <IP address> 

This may be because the process has not completed previous shutdown activities. Please wait and retry again. 

Database start up failed. Processes still running. 

Press RETURN to continue 

Environment

OS: Red Hat Linux 6.x

Resolution

This error would be caused by ungraceful previous shutdown.  Please try following (1).  If it is not still solved, try (2).

Note: 

• The 'dradmin' account is the default database user.  Please use your database admin account if you changed it.
• The adminTools can be executed from any node of Data Repository cluster.

 

(1) Simple method

# su - dradmin 

$ /opt/vertica/bin/adminTools 

Select Option "5 Restart Vertica on Host"

Select the problem node and click [OK]

Exit adminTools and dradmin user

$ exit 

 

(2) Forceful method

# su - dradmin 

$ /opt/vertica/bin/adminTools 

Select Option "7 Advanced Menu"

Select SubOption "3 Kill Vertica Process on Host"

Select the problem node and click [OK]

Exit adminTools and dradmin user

$ exit 

With root user run following command on all Data Repository boxes: 

# /etc/init.d/spreadd stop

Please confirm SPREAD process is not running with below.  

# ps -ef|grep spread 

Restart SPREAD process on all Data Repository boxes: 

# /etc/init.d/spreadd start 

Restart Data Repository process:  

# su - dradmin 

$ /opt/vertica/bin/adminTools 

Select Option "5 Restart Vertica on Host"

Select the problem node and click [OK]

Select Option "1 View Database Cluster State" to confirm all Data Repository nodes are UP state

Exit adminTools 

Additional Information

You can also able to confirm the Data Repository status by following command:

# su - dradmin 

$ /opt/vertica/bin/adminTools -t list_allnodes