Data Repository Upgrade Fails with "the vertica process for the database is running"

Products

• Network Observability

Issue/Introduction

During a Data Repository (DR) upgrade in a multi-node cluster, the dr_install.sh script may fail on one or more nodes after succeeding on others. When attempting to start Vertica via adminTools, the following error is displayed:

Error - Empty content returned from host {NODEIP} ... error_message=Error parsing content from remote process ... ValueError: Incoming protocol 2.0 does not match protocol 1.0

Environment

DX NetOps Upgrade to 24.3.x

Cause

The installation or startup script fails because a Vertica process from a previous session is still running. This is typically caused by an ungraceful shutdown or a prior failed upgrade attempt that left a process hung. The installer detects this running process and cannot proceed.

Resolution

The hung Vertica process must be identified and manually terminated before the upgrade can be successfully retried.

1. Identify the Hung Vertica Process:

   • Log in to the affected Data Repository node as the dradmin user (or your custom DR admin user).
   • Run the following command to list running Vertica processes:
     bash ps aux | grep vertica
   • Look for the main Vertica process. It will look similar to the example below. Note the Process ID (PID), which is 6995 in this case.
     dradmin 6995 3.4 28.1 23290392 4569376 ? Sl May25 2835:46 /opt/vertica/bin/vertica -D /disk2/catalog/drdata/v_drdata_node0001_catalog ...

2. Terminate the Hung Process:
   You can use the recommended adminTools method or the more forceful manual kill command.

   Method 1: Using adminTools (Recommended)

   • As the dradmin user, launch the Vertica administration tool:
     bash /opt/vertica/bin/adminTools
   • From the main menu, select 7) Advanced Menu.
   • From the Advanced Menu, select 3) Kill Vertica Process on Host.
   • Select the node where the hung process is running and click [OK].
   • Exit adminTools.

   Method 2: Manual Kill Command (Forceful)

   • If adminTools is unresponsive or fails, use the Linux kill command with the PID found in step 1.
     bash kill -9 <PID>
     For example: kill -9 6995

3. Retry the Upgrade:
   After confirming the hung process is stopped, re-run the dr_install.sh upgrade script. The installation should now proceed without error.

Before Killing the process - Please make sure you have a valid backup of vertica database which was taken as per the documentation using vbr.py script.