DR DB fails to start after an upgrade or outage

Products

• Network Observability
• CA Performance Management

Issue/Introduction

When the DR DB is restarted, through an upgrade or other maintenance activities like system patching, it may return the following symptoms.

DR DB fails to start with "Unable to read database catalogs" message due to expired password for dradmin user.

 

During a DR DB Vertica upgrade the dr_install.sh fails. In it's log file we see:

Ensuring all nodes updated with vertica-25.3.0-8.x86_64.................[OK]

===============================================================================

Starting Vertica database

===============================================================================

The following hosts could not be reached by SSH and will not be started:

v_drdata_node0002 (IP_Address) will be omitted

v_drdata_node0003 (IP_Address) will be omitted

 

Going with traditional slower startup

Starting nodes: 

v_drdata_node0001 (IP_Address)

Error: Quorum not satisfied for drdata.

1 < minimum 2  of 3 primary nodes.

Attempted to start the following nodes:

Primary

v_drdata_node0001 (IP_Address)

Secondary

 

Database drdata did not start successfully: Cluster partitioned. 

Hint: if you use Eon mode, then you may need to start all primary nodes.

Starting Vertica database failed........................................[FAIL]

 

 

If we try to start vertica from adminTools, option 5, restart vertica on host, it may returns to screen with nothing happening.

 

If a multi-node cluster has only one node with dradmin user password expired, the other nodes may start while it waits for the remaining node to be started.

 

When using the cmd line force option:

./admintools -t restart_node -F -s IP-of-node -d drdata

We get an error to the CLI that states:

Info: no password specified, using none
*** Restarting nodes for database drdata ***
 Restarting host [IP-address] with catalog [v_drdata_node0001_catalog]
Unable to read database catalogs - cannot start database.
Restart Nodes result:  Unable to read database catalogs

 

If we look in the /opt/vertica/log/adminTools.log file we may see the following message:

<Date><Time> admintools/188457:0x7f3c0672f740 [AdapterConnectionPool_3.connect2_using_stored_auth] <WARNING> Unable to SSH as dradmin to <IP_ADDRESS>: UNEXPECTED EOF: Could not login with ssh. ssh output: Your account has expired; please contact your system administrator\nAuthentication failed.\n"

 

Running "ssh ls dradmin@<Node_IP>" from node to node reveals that select nodes are not able to successfully communicate with each other using the required passwordless SSH configurations that are set up and valid.

Environment

All supported DX NetOps Performance Management releases

Cause

The OS dradmin user, the DB admin user, has an expired password at the OS level.

Resolution

To determine which nodes have a dradmin user that requires a password reset follow these steps.

1. Log in to each node as the root user.
2. Run "chage -l dradmin".
   • Substitute the default recommended user name dradmin with your name if different.

For each user with an expired account due to an expired password reset the password to reenable the user account.

Make sure that passwordless ssh is enabled and working for the dradmin OS user to the local node and ALL other nodes in the vertica cluster.

See the KB article Changing dradmin OS user password in Performance Management for additional information on resetting the dradmin OS users password.