Aria Operations for Logs is down.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• The Aria Operations for Logs UI is unresponsive via the cluster VIP FQDN.
• The node(s) are accessible via SSH.
• Cassandra is unstable and all nodes show as down (DN) or 'Connection refused' when running 'nodetool-no-pass status'.
• Running the command below shows that the 'messages' file exceeds 4 GB

du -h /var/log/ | sort -h

Environment

Aria Operations for Logs 8.x

Cause

The root partition is full on one or more nodes in the cluster. This can be checked by running the following on each node in the cluster:

df -h

Note: Typically, the root partition is either sda4 or sda5.

Resolution

Before making any changes to the appliance, it is recommended to ensure snapshots are taken.

• How to take a snapshot of Operations for Logs

Please follow the steps below to truncate the 'messages' file;

• Stop all loginsight services on all nodes;

service loginsight stop

• The '/var/log/messages' file can be truncated by running the following commands:

cd /var/log/
truncate -s 0 messages

or

cat /dev/null > /var/log/messages:

• Following running these commands, please validate storage utilisation;

df -h

• The boot partition should be around 40 - 45%.In some cases, the utilisation may not reflect the change and remains at 100%a for this reason we suggest a reboot of all nodes in either case.

reboot -f

Note: If you find many files with a 0 byte size, you may delete these as well by navigating to the appropriate directory and running:

find . -maxdepth 1 -empty -delete