100% full /storage/core Aria Operations for Logs.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• One of more nodes in the Aria Operations for Logs cluster is showing that the /storage/core disk partition is at 100% usage when running the command df -h on root ssh session.
• If all nodes are affected you may experiencing an issue with accessing the UI
• If some of the nodes are affected with storage/core full you may experience the error "Sorry something went wrong" when exploring logs , you may also receive the error "no permissions" when refreshing the UI or the following error "Error authenticating user" when logging in.
• Aria Operations for logs node(s) may show as as disconnected when checking the cluster status by navigating to Management > Cluster
• You may find the following errors in the /storage/core/loginsight/var/cassandra.log indicating an issue with connection to Cassandra

INFO  [Messaging-EventLoop-3-14] 2025-08-12T09:08:16,793 NoSpamLogger.java:105 - /###.###.###.###:7000->/###.###.###.###:7000-URGENT_MESSAGES-[no-channel] failed to connectio.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /###.###.###.###:7000
Caused by: java.net.ConnectException: Connection refused

• You may find the following errors in the /storage/core/loginsight/var/cassandra.log which indicate an issue with space:

WARN  [CompactionExecutor:4] 2025-03-27T09:52:11,040 CompactionTask.java:92 - insufficient space to compact all requested files. 0.019885063MiB required, BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-............../nb-3849-big-Data.db'), BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-.................-big-Data.db'), BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-..............-big-Data.db'), BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-...........-big-Data.db'), BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/sstable_activity-..........Data.db') for compaction.......... - removing largest SSTable: BigTableReader(path='/storage/core/loginsight/cidata/cassandra/data/system/

 

Environment

Aria Operations for logs 8.18.x

Cause

A node can be running out of disk due:

High ingestion rate (event per second - eps) can cause the node to fill up it's disk faster than normal.

 

Resolution

If the /storage/core is 98%,99%,100%

• Take snapshots without memory for the Aria Operations for Logs 
• Increase the Storage Capacity of the VMware Aria Operations for Logs Virtual Appliance

If the size of the disk is beyond 4tb , Scale out the cluster by adding more nodes. Check VMware Aria Operations for Logs Configuration Limits

• Adding a node to the cluster

To review ingestion rate, (event per second / eps )

• Go to Management-System Monitor - Statistics- Events ingestion Rate.This document: Sizing the VMware Aria Operations for Logs Virtual Appliance  can be a guide to understand If the node(s) are undersized.

For example If the node(s) are mediums size, the max eps for that size is 5000, but via  Management-System Monitor - Statistics- Events ingestion Rate is 6000 constantly, that is a good indicative the nodes need to be change to large size.

-Power off the node in VC

-Edit the settings

-Change to desire size.

-Power on the node

 

Additional Information

VMware Aria Operations for Logs Runs Out of Disk Space