The Aria Operations for Logs (Formerly vRealize Log Insight) root partition is full

Products

VMware Aria Suite

Issue/Introduction

This article provides steps to reduce the sizes of audit.log and auth.log-xxxxxxx files

Symptoms:

You are unable to connect to Log Insight as services are not started, or the virtual machine hangs on boot.
Upon opening the console to the virtual machine, you notice the message: Network Service has failed to start.
You notice high capacity used on /dev/sda-x (where x is typically 4 or 5).

Note: Run this command to check disk-space:
df -h

You notice high disk usage on these two directories: /var/log and /var/log/audit

Note: Run this command to check disk usage on specific directories:

du -hc /var/log && du -hc /var/log/audit | sort -n

Environment

Aria Operations for Logs 8.x

VMware vRealize Log Insight 8.x

Cause

The /var/log and /var/log/audit directories consumes majority of the space on /dev/sda-x (where "x" is typically 4 or 5).
This issue can occur due to excessive logon attempts from a network scanner or vulnerability scanner.

Resolution

Workaround:
To workaround the issue, boot into Single User mode to clear the filled log files, and configure log rotation.

In the vSphere Client, open the console of the desired vRealize Log Insight node.
With the console open, restart or power on the virtual machine.
When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or linux for new 8.x deployments.

Notes:

Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
Type the letter e to go to the GNU GRUB edit menu.
The cursor appears at the end of a line of boot options near the bottom of the display.
If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.

At the end of the line, add a space, then type rw init=/bin/bash which adds another option to the line.
Press F10.

Note: The virtual appliance starts in single-user mode.

Run the following commands to delete the audit.log and auth.log-xxxxxxx files.

rm /var/log/audit/audit.log
rm /var/log/auth.log*

Exit Single User mode and boot the virtual machine normally

reboot -f

Log into the node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
Open /etc/audit/auditd.conf in a text editor and set the max_log_file_action value to ROTATE, then save and close the file.

Note: Skip this step on vRealize Log Insight 8.4 and later.

Run the following command to create the auth-logrotate file:

Note: Skip this step on vRealize Log Insight 8.1 and later.

touch /etc/logrotate.d/auth-logrotate

Open /etc/logrotate.d/auth-logrotate in a text editor and add the following content, then save and close the file:

Note: Skip this step on vRealize Log Insight 8.6 and later.

/var/log/auth.log {
daily
missingok
rotate 5
compress
delaycompress
notifempty
create 640 root root
}

Open the logrotate file in a text editor.

Notes:

For vRealize Log Insight 8.4.1 and earlier the path is /etc/cron.daily/logrotate.
For vRealize Log Insight 8.6 and later the path is /etc/cron.hourly/logrotate.

Before the last line, add the following content then save and close the file:

if [[ -f /var/log/auth.log && ! -s /var/log/auth.log ]]; then
systemctl restart rsyslog
fi

Example: After editing, the file should look similar to the following.
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
if [[ -f /var/log/auth.log && ! -s /var/log/auth.log ]]; then
systemctl restart rsyslog
fi
exit $EXITVALUE

Additional Information

Aria Operations for Logs (Formerly vRealize Log Insight) 8.6 and higher contain a fix to address the log rotation issues however this issue may still occur due to excessive logins from network and vulnerability scanners.