/storage/core file system shows 100% utilization in Aria Operations for Logs.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• One or more nodes in the Aria Operations for Logs cluster may report /storage/core at 100% utilization when executing the df -h command from an SSH root session.
• If all nodes in the cluster are affected, users may experience issues accessing the UI.
• Unable to access Aria Operations for logs UI using VIP or ILB(Internal Load Balancer).
• You may receive “Sorry, something went wrong” error while viewing the Explore Logs section.
• “No permissions” error is noticed when accessing the Aria Operations for logs UI.
• "Error authenticating user” error is noticed while trying to login to Aria Operations for logs UI.
• Aria Operations for logs node(s) may show as as disconnected when checking the cluster status by navigating to Aria Operations for logs UI > Management > Cluster.
• You may find the following errors in the /storage/core/loginsight/var/cassandra.log indicating an issue with connection to Cassandra:
  

  INFO  [Messaging-EventLoop-3-14] ####-##-###:##:### NoSpamLogger.java:105 - /###.###.###.###:7000->/###.###.###.###:7000-URGENT_MESSAGES-[no-channel] failed to connectio.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /###.###.###.###:7000
  Caused by: java.net.ConnectException: Connection refused

• You may find the following errors in the /storage/core/loginsight/var/cassandra.log which indicate an issue with space:
  

  WARN  [CompactionExecutor:4] ####-##-###:##:### CompactionTask.java:92 - insufficient space to compact all requested files. 0.019885063MiB required, BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-............../nb-3849-big-Data.db'), BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-.................-big-Data.db'), BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-..............-big-Data.db'), BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-...........-big-Data.db'), BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-..........Data.db') for compaction.......... - removing largest SSTable: BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/

• Executing the 'nodetool-no-pass status' command on the node with /storage/core at 100% utilization returns the error "Cassandra is not running".
• Running the same command on unaffected nodes confirms the status of the impacted node is explicitly reported as Down.

Environment

Aria Operations for logs 8.18.x

Cause

The node ran out of disk space due to high ingestion rates (Events Per Second - EPS) rapidly increasing storage consumption and exhausting the available capacity in the /storage/core partition.

Resolution

If /storage/core utilization exceeds 97%, perform the following actions:

• Increase the size of the /storage/core partition by following the steps mentioned in the below KB:
  • Reference Link: Increase the Storage Capacity of the VMware Aria Operations for Logs Virtual Appliance
• If the disk size already exceeds 4 TB, scale out the cluster by adding additional nodes. Refer to the technical documentations:
  • VMware Aria Operations for Logs Configuration Limits
  • Adding a node to the cluster

 

 

Additional Information

To review ingestion rate, (event per second / eps )

• Navigate to Management > System Monitor > Statistics- Events ingestion Rate. 
• Compare the observed EPS against the supported limits for the node size in the below documentation.
  • Reference Link: Sizing the VMware Aria Operations for Logs Virtual Appliance

For example:

• • Medium-sized nodes support up to approximately 5000 EPS
  • If the environment consistently ingests around 6000 EPS, the nodes should be upgraded to a larger size.

Steps to resize the node:

1. Power off the node in vCenter
2. Edit the virtual machine settings
3. Allocate resources in the node to the desired size
4. Power on the node again

• Navigate to System Monitor > Statistics to verify the ingestion rate.