Upgrade to Aria Operations for Logs 8.14 does not proceed after primary node

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• The upgrade to Aria Operations for Logs 8.14 does not proceed to the worker nodes after completing the primary node.
• Running the below command via SSH or console to the primary node, as root, returns 127.0.0.1 as IP address of one of the nodes

nodetool-no-pass status

Environment

VMware Aria Operations for Logs 8.14.x

Resolution

This is a known issue affecting VMware Aria Operations for Logs (formerly known as vRealize Log Insight) 8.14, there is no resolution at this time.  Please subscribe to this article to be informed when updates are published.

Workaround:
To workaround the issue, perform the below steps with the upgrade in the incomplete state (do not revert to snapshots prior or run these steps on 8.12):

1. Download the attached file, fix_hostname.tar.gz, from the article
2. Using an SCP Utility like WinSCP or Filezilla, copy the fix_hostname.tar.gz file to the /tmp directory on the Primary node.
3. SSH to the Primary node as root
4. Change directory to /tmp

cd /tmp

5. Run the following command to extract the script from fix_hostname.tar.gz

tar zxf fix_hostname.tar.gz

6. Run the following command to execute the script

bash fix_hostname.sh
 

Note: You should see an output of "Done" from the script indicating it has completed

7. Restart the Log Insight service starting with the Primary node, and then on all remaining nodes in the cluster

service loginsight restart

Note: The upgrade should now continue automatically to the other nodes of the cluster. If not, perform the steps outlined in How to Manually Complete a VMware Aria Operations for Logs Upgrade (92130)

8. After the upgrades complete, navigate to Administration > Cluster and note the uptime values for all nodes.  They should have similar uptimes, indicating all nodes have been restarted simultaneously after the upgrade.

Note: If the nodes do not show the same uptime, SSH or console to them each as root, and restart the service simultaneously on all nodes with the below command

service loginsight restart

9. In a browser, navigate to https://VRLI_IP_OR_FQDN/internal/config

Note: Replace VRLI_IP_OR_FQDN in the above command with the IP or Fully Qualified Domain Name of your Aria Operations for Logs primary node

10. Tick the "Show all settings" checkbox and search for the "secure-rpc" and "authenticated-rpc" settings. The values for both of these settings should be set to true.  If not, manually change them to true, and click Save.

Note: An automatic service restart of all nodes should occur within ~5 minutes.  The status of which can be observed from the Administration > Cluster page.  After 5 minutes, if you see higher uptimes than when the above change was made, restart the service simultaneously on all nodes with the below command

service loginsight restart 

Confirming these settings ensures the cluster is properly secured from VMSA-2023-0021

Additional Information

Impact/Risks:
This issue prevents subsequent nodes in a VMware Aria Operations for Logs (formerly known as vRealize Log Insight) cluster from upgrading after the primary node, resulting in an incomplete upgrade.