VMware Aria Operations for Logs 8.14 New Worker Node is Disconnected Shortly After Addition to Cluster

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

1. After following Add a Worker Node to a VMware Aria Operations for Logs Cluster , the worker node displays Connected in the Management > Cluster page for a short time then changes to Disconnected.
2. In a SSH session logged on to the primary VMware Aria Operations for Logs node as the root user, the output of ls -l /usr/lib/loginsight/application/etc/certs/cluster.pem shows a 0-byte file size for the cluster.pem file.

Environment

VMware Aria Operations for Logs 8.14.x

Cause

A corruption of the certificate that is distributed to new worker nodes during the addition to a VMware Aria Operations for Logs cluster occurs during some upgrades.

Resolution

To resolve the issue, complete the steps below to re-add the disconnected worker node to the VMware Aria Operations for Logs cluster.

Quick Links:

• Remove Disconnected Worker Node
• Download Certificate
• Re-add Worker Node

 

Remove Disconnected Worker Node

1. Follow the steps from Remove a Worker Node from a VMware Aria Operations for Logs Cluster to remove the disconnected worker node from the VMware Aria Operations for Logs cluster.

Note: The worker node with disconnected status cannot be placed into maintenance mode. Step 2. from the procedure can be ignored.

 

Extract Certificate

1. Download the extract_cert.zip file attached to this article from the Attachments section on the right.
2. Using an SCP utility like WinSCP, copy the extract_cert.zip file to the /root directory on the Primary node.
3. Log into the Primary node as root via SSH or Console.
4. Run the following command to change to the /root directory:

cd /root

5. Run the following command to extract the script from extract_cert.zip:

unzip extract_cert.zip

6. Run the following command to update the permissions of the script:

chmod 744 extract_cert.sh

7. Run the following command to run the extract_cert.sh script:

./extract_cert.sh
 

Note: The script will create custom.pem in the /root directory. You should see output similar to

:

8. Run the following command to repair the cluster.pem file:

cat ./custom.pem > /usr/lib/loginsight/application/etc/certs/cluster.pem

 

 

Re-add Worker Node

1. Log in to the worker node that was removed from the VMware Aria Operations for Logs cluster as root via SSH or Console.
2. Run the following command to stop loginsight service:

systemctl stop loginsight

3. Run following command to return the node to bootstrap mode:

rm /storage/core/loginsight/config/*

4. Run the following command to start loginsight service:

systemctl start loginsight

Note: Allow 5 minutes for the service to completely start before proceeding to the next step.

5. Use the steps from Join an Existing Deployment to re-add the worker node to the VMware Aria Operations for Logs cluster.