Replace a node in an Aria Operations for Logs cluster

Products

• VMware Aria Suite

Issue/Introduction

This article provides steps to replace a Worker node with a new node. This can be used as a workaround if a node is experiencing issues.
Note: If the Primary node needs to be replaced, engage Broadcom support to assist as the procedure is different.

Environment

Aria Operations for Logs 8.x and later

Resolution

It is possible to deploy a new node, and copy the data buckets from the Old node to the New node. Follow this procedure:

1. Take a  snapshot of the Aria Operations for logs nodes before you proceed with the below steps.
2. Remove the problematic Worker Node from the cluster. Do not delete the VM as you will need it to copy the historical data from it.
3. Add a new Worker Node to the cluster. Make sure to add storage capacity to this node if needed, to match the other nodes of the cluster.
4. Log in as root (SSH or Console) and stop the service on both the Old node and New node, using this command:
   service loginsight stop
5. Copy the historical data from the Old node to the New node, using this command (replace newnode_IP by the IP of the new node):
   scp -r /storage/core/loginsight/cidata/store newnode_IP:/storage/core/loginsight/cidata 
6. On the New node, run the following command to import the logs from the data buckets:
   for bucket in $(ls /storage/core/loginsight/cidata/store | grep -v 'generation\|buckets\|strata_write.lock'); do echo y | /usr/lib/loginsight/application/sbin/bucket-index add $bucket --statuses archived; done
7. Start the service on the New node with this command:
   service loginsight start
8. Once verified that the cluster is working as expected, you can remove the pre-procedure snapshot, and delete the Old VM.

Note: It is possible to keep the same IP/FQDN for the new node. The historical data would need to be copied to a different temporary location, and the Old node powered off before deploying the New node.