Aria Operations for Logs UI unresponsive due to full /storage/var partition caused by Catalina log growth

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• The Aria Operations for Logs UI displays a "Page not found" error, making it inaccessible to users.
• Users are unable to access or interact with the UI due to the service being unresponsive.
• The /storage/var partition has reached 100% utilization, primarily caused by large Catalina logs accumulating in the /storage/var/loginsight/apache-tomcat/logs directory.
• The /storage/core/loginsight/var/runtime.log file contains the following error, indicating that node is not available:

  com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query

• When checking the Cassandra node status using the command: /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool-no-pass status , it shows that one or more nodes are down.

 

 

Environment

Aria Operations for Logs 8.18.x

Cause

• Rapid growth of the Catalina logs causes this issue, which results in the /storage/var partition becoming full.
• Log growth is caused by repeated Tomcat session loading failures when Cassandra is down.
• Cassandra service becomes unavailable, preventing Tomcat from retrieving session data from the database.
• This leads to constant logging of session load failures, which quickly fills the catalina.log, ultimately making the UI inaccessible.

Resolution

To resolve the issue and restore UI functionality, follow these steps in order:

 

NOTE: Kindly take snapshots on all the Aria Operations for logs nodes before performing the below operations.

1. Stop the Log Insight service using the command: 

   service loginsight stop

2. Clear excessive Catalina logs:

   • Navigate to the Catalina log directory: /storage/var/loginsight/apache-tomcat/logs

   • Remove or archive large log files to free up space: rm -f catalina.<YYYY-MM-DD>*.log or delete only the log which is occupying huge space.

   • If the working log file (catalina.log) is consuming excessive disk space, it is not recommended to delete the file. Instead, you can safely clear its contents by running the following command:

     echo > catalina.log

3. Start the Log Insight service:

   service loginsight start

4. Verify Log Insight service status:

   • After the restart, check if the Log Insight processes are running:

     ps aux | grep loginsight

5. Check Cassandra node status:
   • Verify the Cassandra status on all nodes using the following command:

     /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool-no-pass status

   • Ensure that the status is "UN" for all nodes.
   • If any node is not up, it is recommended to contact Broadcom Support for further assistance.