Data Repository nodes with different Vertica version from failed upgrade

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

An upgrade to the DX NetOps Performance Management (PM) Data Repository (DR) Vertica database was run.

After running the dr_install.sh script it failed to complete the upgrade. It did update the Vertica rpm on the node the dr_install.sh was run from. When it failed to communicate with the other nodes in the database it didn't update the rpm on those nodes.

As a result further dr_install.sh runs fail due to messages about different versions. These are the messages seen from the dr_install.sh run that indicate the problem.

*** Starting database: drdata ***
* Database config synchronization step failed. Please check logs *
        Starting nodes:
                v_drdata_node0001 (x.x.x.x)
                v_drdata_node0002 (y.y.y.y)
                v_drdata_node0003 (z.z.z.z)
Error: Vertica versions do not match on all database hosts. Startup cannot continue.
Host: Node_2_IP Version: Vertica Analytic Database v9.1.1-5
vertica(v9.1.1-5) built by jeeves@build-centos6 from tag@releases/VER_9_1_RELEASE_BUILD_1_5_20181108 on 'Thu Nov  8 12:25:31 2018' $BuildId$
Host: Node_2_IP Version: Vertica Analytic Database v9.1.1-5
vertica(v9.1.1-5) built by jeeves@build-centos6 from tag@releases/VER_9_1_RELEASE_BUILD_1_5_20181108 on 'Thu Nov  8 12:25:31 2018' $BuildId$
Host: Node_1_IP Version: Exception encountered during verion check: [Could not find acknowledgement of remote call.]
Host: Node_1_IP Version: Exception encountered during verion check: [Could not find acknowledgement of remote call.]
Host: Node_1_IP Version: Exception encountered during verion check: [Could not find acknowledgement of remote call.]
Host: Node_3_IP Version: Exception encountered during verion check: [Could not find acknowledgement of remote call.
Database start up failed.  Version mismatch
Press RETURN to continue

When checking the versions for the different nodes we see that they have different versions of the Vertica rpm.

[dradmin@v_drdata_node0001 bin]$ rpm -qa | grep vertica
vertica-10.1.1-0.x86_64
 
[dradmin@v_drdata_node0002 bin]$ rpm -qa | grep vertica
vertica-9.1.1-5.x86_64
 
radmin@v_drdata_node0003 bin]$ rpm -qa | grep vertica
vertica-10.1.1-0.x86_64

Environment

All supported DX NetOps Performance Management releases

Cause

This was due to the drinstall.properties file not being filled in correctly. In this instance the DbHostNames variable was not populated with the cluster node names.

Resolution

Ensure the drinstall.properties file is properly edited. Ensure the nodes are able to communicate with each other.

Then re-run the dr_install.sh script.