Data Repository Node Stuck in Recovering after a planned stop

Products

• CA Infrastructure Management
• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

We shutdown a node in the DR cluster for maintenance but when bring the node back up it did not start up so used the below command and now its stuck in recovering.

 

/opt/vertica/bin/admintools -t restart_node -d drdata --hosts x.x.x.x --force

Environment

Release : 3.x

Component : IM Data Storage

Cause

If a transaction is in queue or in progress when the node is shut down, the LGE (Last Good Epoch) may not match the other nodes when the down node restarts.

Resolution

This is working as designed.

The whole point of having 3 or more nodes in the vertica cluster is to allow k-safety.

This process will recover or rebuild a node that was down when that node restarts.