The Explore Logs search page fails to render incoming data streams or run filtering queries, throwing a red banner exception: Sorry there was a problem with your search

Products

VCF Operations

Issue/Introduction

Navigating to Aria Operations for logs UI > Management > Cluster page > Cluster presents a degraded view reflecting only a single node accompanied by the exception string: "internal error occurred during request processing".

Running the df -h command through an SSH session on all nodes shows the root (/) partition reached 100% utilization on affected nodes.
Additionally, the /boot partition utilization was found to be above 80% across all nodes in the cluster.
Running the nodetool-no-pass status command through an SSH session on all nodes shows DN for few nodes in the cluster.

Environment

Aria Operations for logs 8.18.x

Cause

The issue was caused by a cluster-wide persistence layer failure due to filesystem capacity exhaustion on multiple VMware Aria Operations for Logs cluster nodes.

The filesystem utilization increased significantly, causing the affected partitions to reach full capacity. Once the storage volumes were exhausted, the Linux operating system transitioned the filesystem into a read-only state as a protection mechanism to prevent filesystem corruption.

Due to the read-only filesystem state, the Cassandra-based persistence layer was unable to perform required database write operations. This impacted communication between cluster nodes and caused failures in Cassandra data synchronization.

As multiple storage components became unavailable, the cluster lost the required quorum state, resulting in application-level failures and Internal Server Error messages.

The issue was resolved after restoring filesystem availability and recovering the cluster services.

Resolution

Step 1: Cluster Pre-check and Triage Diagnostics

Establish a Secure Shell (SSH) connection to the Primary and Worker nodes using administrative root credentials.
Check the local storage layout across all partition targets:
```
df -h
```
Identify if any node exhibits full disk saturation (100% space utilization) on root or boot partition.
Inspect the health state of the underlying distributed storage ring elements:
```
nodetool-no-pass status
```
Verify if slice states reflect errors. A normal operating ring displays a status of UN (Up Normal) for all members; a status of DN (Down Node) confirms database synchronization dropouts.

Step 2: Clear Active Hypervisor Latencies

Open the vCenter Server inventory console and inspect the virtual machines backing the Aria Operations for Logs cluster.
Identify and resolve any active or legacy virtual machine snapshot delta loops. Consolidated or dangling snapshots reduce disk IO line rate performance and must be removed or cleaned up prior to rebuilding database consistency.

Step 3: Stop Services and Reclaim Disk Space

Execute a graceful operating system shutdown across all cluster members and power off the virtual appliances to protect database index tracking blocks from runtime metadata corruption.
Power on the impacted nodes one by one to begin immediate space reclamation following the guidelines in Broadcom KB 375186.
Stop the application daemon service on the active node:
```
systemctl stop loginsight
```
Truncate the bloated log files to empty the storage space without breaking open active tracking descriptors:
```
truncate -s 0 /var/log/messages
truncate -s 0 /var/mail/root
```
Inspect the boot partitions using df -h. If older kernel files are filling up space, clear out old deployment configurations according to Broadcom KB 378978 to keep general boot partition allocation strictly below 80%.

Step 4: Recover and Resynchronize the Cassandra Database

Once storage capacity is successfully recovered across all nodes, align the database cluster nodes using the database runbooks outlined in Broadcom KB 389806:

Ensure the core loginsight application service remains stopped across all appliances:
```
systemctl stop loginsight
```
Manually force the standalone Cassandra process to execute initialization threads on each node:
```
/usr/lib/loginsight/application/sbin/li-cassandra.sh --startnow --force
```
Monitor the status of the ring array to confirm all nodes have re-established contact and transitioned back into an up status:
```
nodetool-no-pass status
```

If a node fails to show UN, force a recycle on that specific instance:

/usr/lib/loginsight/application/sbin/li-cassandra.sh --stopnow --force
/usr/lib/loginsight/application/sbin/li-cassandra.sh --startnow --force

Once all members reliably display a status of UN, initiate a disk flush and execution block repair routine to reconcile data tables across the cluster ring:
```
nodetool-no-pass flush
nodetool-no-pass repair
```
After the database repair completes successfully, stop the standalone Cassandra threads and start the main application daemon on each node. Ensure each node is fully online before proceeding to the next instance:
```
/usr/lib/loginsight/application/sbin/li-cassandra.sh --stopnow --force
systemctl start loginsight
```

Step 5: Post-Remediation Verification

Log into the Aria Operations for Logs configuration interface.
Navigate to Management > Cluster and confirm the structural configuration map accurately tracks all 5 nodes in a healthy Connected state.
Access the Explore Logs workspace window and execute a series of validation searches to ensure data indexes load correctly without throwing exceptions.