The Aria Operations for Logs root partition is full

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article provides steps to reduce the sizes of audit.log and auth.log-######## files. And remove java_pid####.hprof files.

Symptoms:

One or more nodes are disconnected in the cluster.
You see the following log traces in /var/log/vmware/loginsight/cassandra.log:

Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/storage/core/loginsight/cidata/cassandra/data/machine_learning/spock_cluster_counts-#################/nb-#####-big-Data.db): corruption detected, chunk at 293732 of length 29154.
You are unable to connect to Log Insight as services are not started, or the virtual machine hangs on boot.
Upon opening the console to the virtual machine, you notice the message: Network Service has failed to start.
Aria Operations for Logs Service Fails to Start on Boot

ERROR : Failed to start VMware Aria Operations for Logs
You need to access the virtual IP (VIP) address, but there's no connection
You notice high capacity used on /dev/sda-x (where x is typically 3, 4 or 5).

Note: Run this command to check disk space:

df -h

You notice high disk usage on these directories: /usr/lib/loginsight, /var/log, or /var/log/audit

Note: Run this command from the / (root) partition and keep going to the largest directory, then you may find it is /usr/lib/loginsight utilizing a large amount. Here you may find one or more java_pid####.hprof large files in this directory.

du -hscx * 2>/dev/null | sort -h

Note: Run the following command to check disk usage on the specific directories:

du -hc /var/log && du -hc /var/log/audit | sort -n

Environment

Aria Operations for Logs 8.x
Aria Operations for Logs 8.18.x
VMware vRealize Log Insight 8.x

Cause

The /var/log, /var/log/audit or /usr/lib/loginsight directories consumes the majority of the space on /dev/sda-x (where "x" is typically 3, 4 or 5).
This issue can occur due to excessive login attempts from a network scanner or vulnerability scanner.
hprof file is generated due to service crashes, which seem to be a trending issue when the cluster is running low on live storage and no data archiving is enabled, or when the cluster is undersized.

NOTE: In newer versions such as 8.18.0 and above, it's typically not /var/log as the issue but more often due to hprof files in /usr/lib/loginsight

Resolution

Workarounds:

Part 1. Issue due to `hprof` file(s)

SSH into the node as root.

NOTE: If you are not able to connect as root due to an expired password and attempts to reset are not working. Follow steps 1-5 below to Boot into Single User mode then come back to step 2
Go to /usr/lib/loginsight and remove the hprof file(s) by running the following commands:

NOTE: If you need to save hprof files for analysis or other reasons, archive them to another remote location prior to deleting

cd /usr/lib/loginsight
rm java_pid####.hprof

** At this point, if you had to boot into single user mode due to a root password reset issue, run passwd root command now to reset it.
Reboot the node with reboot -f (if in single user mode) or restart the loginsight service by running the command:

service loginsight restart
Repeat the above steps on any other nodes that have 100% full root partition.

Part 2. Issue due to `/var/log, /var/log/audit` being the largest directories.

Boot into Single User mode to clear the filled log files, and configure log rotation.

In the vSphere Client, open the console of the desired Aria Operations for Logs node.
With the console open, restart or power on the virtual machine.
When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.

Notes:

- - Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
  - Type the letter e to go to the GNU GRUB edit menu.
  - The cursor appears at the end of a line of boot options near the bottom of the display.
  - If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.

At the end of the line, add a space, then type rw init=/bin/bash which adds another option to the line.
Press F10.

Note: The virtual appliance starts in single-user mode.

Run the following commands to delete the audit.log and auth.log-####### files.

rm /var/log/audit/audit.log
rm /var/log/auth.log*

Exit Single User mode and boot the virtual machine normally

reboot -f

Log into the node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
Open /etc/audit/auditd.conf in a text editor and set the max_log_file_action value to ROTATE, then save and close the file.

Note: Skip this step on vRealize Log Insight 8.4 and later.

Run the following command to create the auth-logrotate file:

Note: Skip this step on vRealize Log Insight 8.1 and later.

touch /etc/logrotate.d/auth-logrotate

Open /etc/logrotate.d/auth-logrotate in a text editor and add the following content, then save and close the file:

Note: Skip this step on vRealize Log Insight 8.6 and later.

/var/log/auth.log {
daily
missingok
rotate 5
compress
delaycompress
notifempty
create 640 root root
}

Open the logrotate file in a text editor.

Notes:

- - For vRealize Log Insight 8.4.1 and earlier the path is /etc/cron.daily/logrotate.
  - For vRealize Log Insight 8.6 and later the path is /etc/cron.hourly/logrotate.

Before the last line, add the following content then save and close the file:

if [[ -f /var/log/auth.log && ! -s /var/log/auth.log ]]; then
systemctl restart rsyslog
fi

Example: After editing, the file should look similar to the following.
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
if [[ -f /var/log/auth.log && ! -s /var/log/auth.log ]]; then
systemctl restart rsyslog
fi
exit $EXITVALUE

Additional Information

Aria Operations for Logs (Formerly vRealize Log Insight) 8.6 and higher contain a fix to address the log rotation issues. However, this issue may still occur due to excessive logins from network and vulnerability scanners.