Upgrading the OS on DR host nodes. How can we a node in a cluster to a new node with the upgraded OS?

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

In the process of upgrading the OS of all the CAPM systems. In order to upgrade the OS of the DR cluster, the plan is to build a new node with RHEL 8, to replace an old node with RHEL 6.

The DR cluster is a 3 node cluster. Our plan is to replace node 3, with the new node_3a. Once that has been replaced and synced, then replace node 2 with new node 2a and lastly, replace node 1 with node node 1a.

The above process will result in no outage as the DR cluster can run on 2 nodes?

How do we go about replacing a node in the cluster with a new node? What is the step by step process? 

Environment

DX NetOps CAPM Release : 21.2 or later

Resolution

There's no real support for a node by node, one at a time upgrade. One way is to install a new RHEL8 node, add it to the cluster, rebalance, then remove the original RHE6 node, and then rebalance.   You'll somehow have to do a re-ip from new IP to old IP for each node as you go.   Repeat for the 2 other nodes.

However, this would be painstakingly slow and will affect DA performance the whole time.  Plus all the nodes get renamed from 1,2,3 to 4,5,6 which makes copy-cluster difficult. So we do not recommend this approach.

Instead, we recommend that you follow the 3rd upgrade option suggested in the Vertica KB : 

Vertica KB : Best Practices for Upgrading the Operating System on Nodes in a Vertica Cluster

• Upgrade the Cluster and Restore a Backup

Prior to starting, have backups created as per our TechDocs:

TechDocs : DX NetOps CAPM 21.2 : Back Up the Data Repository

Then update the OS and reinstall the DR then upload the backup. 

Make a backup of the current 3 DR nodes, bring down the DA, then upgrade the OSs on all 3 nodes (if VM's, take a snapshot after shutting down first).  Then once all 3 are upgraded to RHEL8, then install any OS packages specified by our TechDocs:  

TechDocs : DX NetOps CAPM 21.2 : Package Requirements

Then restore the DB from backup so we make sure nothing got corrupted. Start the DA.  If everything runs fine, remove the VM snapshots as snapshots slow down the machine.