The Aria Operations for Logs (Formerly vRealize Log Insight) root partition is full

Products

• VMware Aria Suite

Issue/Introduction

This article provides steps to reduce the sizes of audit.log and auth.log-######## files. And remove java_pid####.hprof files.

Symptoms:

• One or more nodes are disconnected in the cluster.
• You see the following log traces in /var/log/vmware/loginsight/cassandra.log:
  
  Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/storage/core/loginsight/cidata/cassandra/data/machine_learning/spock_cluster_counts-#################/nb-#####-big-Data.db): corruption detected, chunk at 293732 of length 29154.
  
  
• You are unable to connect to Log Insight as services are not started, or the virtual machine hangs on boot.
• Upon opening the console to the virtual machine, you notice the message: Network Service has failed to start.
• Aria Operations for Logs Service Fails to Start on Boot
  
  ERROR : Failed to start VMware Aria Operations for Logs
  
  
• You need to access the virtual IP (VIP) address, but there's no connection
• You notice high capacity used on /dev/sda-x (where x is typically 4 or 5).

Note: Run this command to check disk space:

df -h

• You notice high disk usage on these directories: /var/log, /var/log/audit, and /usr/lib/loginsight.

Note: Run the following commands to check disk usage on specific directories: 

 

du -hc /var/log && du -hc /var/log/audit | sort -n

 

or

 

du -hscx * 2>/dev/null | sort -h

Note: Start this command in / partition and keep going to the largest directory, then you may find it is /usr/lib/loginsight. Then you find one or more java_pid####.hprof large files in this directory.

Environment

Aria Operations for Logs 8.x

VMware vRealize Log Insight 8.x

Cause

• The /var/log, /var/log/audit or /usr/lib/loginsight directories consumes the majority of the space on /dev/sda-x (where "x" is typically 4 or 5).
• This issue can occur due to excessive login attempts from a network scanner or vulnerability scanner.
• hprof file is generated due to service crashes, which seem to be a trending issue when the cluster is running low on live storage and no data archiving is enabled, or when the cluster is undersized.

Resolution

Workaround:

Part 1. Issue due to /var/log, /var/log/audit being the largest directories.

 Boot into Single User mode to clear the filled log files, and configure log rotation.

1. In the vSphere Client, open the console of the desired Aria Operations for Logs node.
2. With the console open, restart or power on the virtual machine.
3. When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.

Notes:

• • • Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
    • Type the letter e to go to the GNU GRUB edit menu.
    • The cursor appears at the end of a line of boot options near the bottom of the display.
    • If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.

4. At the end of the line, add a space, then type rw init=/bin/bash which adds another option to the line.
5. Press F10.

Note: The virtual appliance starts in single-user mode.

6. Run the following commands to delete the audit.log and auth.log-xxxxxxx files.

rm /var/log/audit/audit.log
rm /var/log/auth.log*

7. Exit Single User mode and boot the virtual machine normally

reboot -f

8. Log into the node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
9. Open /etc/audit/auditd.conf in a text editor and set the max_log_file_action value to ROTATE, then save and close the file.

Note: Skip this step on vRealize Log Insight 8.4 and later.

10. Run the following command to create the auth-logrotate file:

Note: Skip this step on vRealize Log Insight 8.1 and later.
 

touch /etc/logrotate.d/auth-logrotate

11. Open /etc/logrotate.d/auth-logrotate in a text editor and add the following content, then save and close the file:

Note: Skip this step on vRealize Log Insight 8.6 and later.

/var/log/auth.log {
daily
missingok
rotate 5
compress
delaycompress
notifempty
create 640 root root
}

12. Open the logrotate file in a text editor.

Notes:

• • • For vRealize Log Insight 8.4.1 and earlier the path is /etc/cron.daily/logrotate.
    • For vRealize Log Insight 8.6 and later the path is /etc/cron.hourly/logrotate.

13. Before the last line, add the following content then save and close the file:

if [[ -f /var/log/auth.log && ! -s /var/log/auth.log ]]; then
  systemctl restart rsyslog
fi

Example: After editing, the file should look similar to the following.
#!/bin/sh
  
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
if [[ -f /var/log/auth.log && ! -s /var/log/auth.log ]]; then
   systemctl restart rsyslog
fi
exit $EXITVALUE

 

Part 2. Issue due to hprof file(s).

1. SSH into the node as root.
2. Go to /usr/lib/loginsight and remove the hprof file(s) by running the following commands:
   
   NOTE: If you need to save hprof files for analysis or other reasons, archive them to another remote location prior to deleting
   
   cd  /usr/lib/loginsight   
rm java_pid####.hprof
   
   
   
3. Reboot the node or restart the loginsight service by running the command:
   
   service loginsight restart
   
   
4. Repeat the above steps on any other nodes that have 100% full root partition.

Additional Information

Aria Operations for Logs (Formerly vRealize Log Insight) 8.6 and higher contain a fix to address the log rotation issues. However, this issue may still occur due to excessive logins from network and vulnerability scanners.