Aria Operations for Logs UI displays intermittent "Servlet Error" during queries

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When users attempt to run queries in the Aria Operations for Logs GUI, the cluster intermittently displays a "Servlet Error." This issue forces users to either log out and log back in or refresh the page to resume operations.

Symptoms

• The UI intermittently fails with a "Servlet Error" during active query sessions.

• The Management > Certificates page may fail to load or show a blank page.

• The cluster may show connection errors in the logs related to the internal database (e.g., Error while opening new channel or Connection refused on port 9042).

• Internal certificate files (e.g., /storage/core/loginsight/cidata/cassandra/config/cacert.pem) may show a file size of 0 bytes.

   

   

Cause

This issue is caused by a synchronization failure or corruption of internal certificates across the cluster nodes, often occurring after a failed certificate replacement or an interrupted upgrade. When the internal certificate file is empty (0 bytes), the UI cannot authenticate with the backend database to retrieve query results, resulting in a servlet error.

Resolution

To reboot an Aria Operations for Logs cluster correctly while maintaining service stability, follow these steps.

Note: In an Aria Operations for Logs (formerly Log Insight) cluster, services should be restarted in a specific order—starting with the Primary node and then proceeding to the Worker nodes.

Prerequisites

• Ensure you have root credentials for all nodes in the cluster.

• Use an SSH client (like PuTTY or Terminal) to access the nodes.

Procedure

1. Log in to the Primary Node

   • Connect to the Primary node via SSH using the root account.

2. Restart the Service on the Primary Node

   • Run the following command to restart the Log Insight service:

     service loginsight restart
     

   • Wait for the command to complete and verify the service status:

     service loginsight status
     

3. Perform a Rolling Restart on Worker Nodes

   • Log in to each Worker node one at a time.

   • Run the restart command on the Worker node:

     service loginsight restart

   • Wait for the service to become fully active on the current Worker node before moving to the next one. This prevents simultaneous downtime across the cluster.

4. Verify Cluster Health

   • Log in to the Aria Operations for Logs web interface.

   • Navigate to Management > Cluster.

   • Verify that all nodes show a status of Connected and that the integrated load balancer (ILB) is functioning correctly.