DR startup process is still running despite the cluster being UP

Products

• CA Infrastructure Management
• CA Performance Management - Usage and Administration

Issue/Introduction

The Linux servers that our DR are running on were patched last night.  This morning I checked and under /opt/vertica/bin/adminTools and the cluster state shows UP.  We checked the processes and the startup Vertica process shows to still be running on 2 of the 3 DR servers.
 

[[email protected]]$ ps -ef | grep rc | grep -v grep
root           1394    1     0 Aug21 ?        00:00:00 /bin/bash /etc/rc.d/rc 3
root           10305   1394  0 Aug21 ?        00:00:00 /bin/bash /etc/rc3.d/S99verticad start

Can we kill this process or is there something else we need to do first?

Environment

Release:
Component: IMSTOR

Resolution

While it is highly likely that killing the process on the others two systems will not cause any problems, being that we are dealing with a database it is better to err on the side of caution.

1. Stop the Vertica cluster through adminTools
2. Run 'ps -ef | grep rc' to see if the rc script is still running
3. If it is still running you can kill -9 its PID
4. Start the Vertica cluster through adminTools

The likely cause is that after patching, the admin team initiated the S99verticad start command on all 3 cluster nodes and as Vertica only needs a start command from one, the other two commands sat lingering.