Unable to log in the Aria Operations for Logs web UI remains stuck loading (spinning) after entering any credentials.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Users are unable to log in to Aria Operations for Logs (vRealize Log Insight) UI using either Active Directory (AD) credentials or the local admin account. After entering credentials, the login screen displays a spinning/circling/loading animation indefinitely or resets to the login screen. The UI may partially load, hanging on the banner. Additionally, commands against the Cassandra database time out.
• Commands against the Cassandra database time out. Example: SELECT COUNT(*) FROM logdb.tomcat_sessions; returns
  ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message "Operation timed out - received only 0 responses." info=['consistency': 'ONE', 'required_responses': 1, 'received_resonses': 0}
• top command via ssh to the nodes show utilization of all cores at 99 and 100% consistently.
• Rebooting the cluster temporarily allows access.
• Events per second or ingestion rate exceed the recommendations in Sizing the VMware Aria Operations for Logs Virtual Appliance, which can be checked while the cluster is accessible a few minutes after the cluster is rebooted.
• The Aria Operations for Logs Sizing Estimator tends to recommend a cluster larger than the 18 node maximum for the ingestion rate and EPS.
• Examining the stats.log file show consistent spikes well over what is supported by the nodes.
• Duplicate log entries show up when Exploring through the logs
• Storage latency is within reasonable thresholds (<1-2ms)

 

Environment

Aria Operations for Logs 8.18.x

Cause

This is caused by overloading the nodes with too many ingestion events, likely because of a misconfigured liagent.ini configured to send everything in the folder, which would likely include the logs that are getting rolled over or changed.

Resolution

Work with the application teams to configure the log insight agent via the liagent.ini file to include only the desired logs.

Additional Information

To help confirm the Log Insight agent is the culprit, enable debugging and restart the agent; examine the top of the agent.log file to see if it is sending entries including log rolling patterns like "/log/location/example.log" and "/log/location/example.log.1"

This can also be used to help troubleshoot what modifications to liagent.ini will result in only the desired logs to be sent.