Performance Management Vertica database fails to start with SSH errors

Products

• CA Infrastructure Management
• CA Performance Management
• Network Observability

Issue/Introduction

The Performance Management Vertica database cluster servers were upgraded to a newer RedHat Kernel version.

After this the Vertica Data Repository database fails to start. It shows the following errors when attempting to start it via the adminTools UI on Node0001.

*** Starting database: <DBName>***
* Database config synchronization step failed. Please check logs *
Could not login with SSH to host <Node2_NameOrIP>
Could not login with SSH to host <Node3_NameOrIP>
        Starting nodes:
                v_drdata_node0001 (<Node1_NameOrIP>)
                v_drdata_node0002 (<Node2_NameOrIP>) ***UNAVAILABLE***
                v_drdata_node0003 (<Node3_NameOrIP>) ***UNAVAILABLE***
The cluster is partitioned with x.x.x.119, x.x.x.120 down.  Over half (2/3) of the nodes must be available.
Database start up failed.  Cluster partitioned.
Press RETURN to continue

Environment

All supported Performance Management releases

Cause

The OS level database administrator user, normally dradmin, has an expired password.

One way to potentially verify if this is the case, is to run the following on each node:

chage -l dradmin

Replace dradmin with the name of the Data Repository administrator username.
This is the same user required to run the adminTools command for database administration

Resolution

Reset the dradmin OS user password