How to clean up excessive Apache Tomcat logs in vRealize Log Insight

Products

• VMware Aria Suite

Issue/Introduction

This article provides steps to reduce the sizes of localhost_access_log.[DATE].txt files

Symptoms:

• You are unable to connect to vRealize Log Insight as services are not started, or the virtual machine hangs on boot
• Upon opening the console to the virtual machine, you notice the message: Network Service has failed to start.
• Follow steps 1-5 of How to reset the root password in vRealize Log Insight (53649) to get into Single User Mode
  • Running the following command, you notice high disk usage on the root partition of /dev/sda-x (where x is typically 4 or 5)

df -h

• Running the following command, you notice high disk usage on the Apache Tomcat Log directory

du -hc /usr/lib/loginsight/application/3rd_party/apache-tomcat-*/logs

Environment

VMware vRealize Log Insight 8.3.x

Cause

This issue is caused by the localhost_access_log.[DATE].txt files not being deleted (or they are rolled, but a large number of old files are kept) which causes a full root partition.

Resolution

This issue is resolved in vRealize Log Insight 8.4.1, which can be downloaded from Broadcom Portal.

Workaround:
To workaround the issue:

1. In single user mode (steps 1-5 of How to reset the root password in vRealize Log Insight (53649), run the following command to remove the excess logs

 

rm /usr/lib/loginsight/application/3rd_party/apache-tomcat-*/logs/localhost_access_log*

2. . Exit Single User mode and reboot the VM

reboot -f

3. SSH to the node as root
4. Using a text editor, open the localhost_access_log configuration file located at /usr/lib/loginsight/application/etc/3rd_config/server.xml
5. Find the line starting with <Valve className="org.apache.catalina.valves.AccessLogValve and add maxDays="4" as the last parameter

Note: After the changes, the entire line should look like this:

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" pattern="%h %{X-Forwarded-For}i %l %u %t &quot;%r&quot; %s %b &quot;%{User-Agent}i&quot;" resolveHosts="false" prefix="localhost_access_log" suffix=".txt" maxDays="4" />

6. Save and exit the file
7. Restart the Log Insight service

service loginsight restart

8. Repeat steps 1-7 on all affected nodes in the vRealize Log Insight cluster.