Aria Operations for Logs UI is inaccessible

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Users are unable to access the Aria Operations for Logs user interface on a designated node(either primary or worker node)
The Cassandra service is crashing on that node, and the following errors are observed in the /storage/core/loginsight/var/runtime.log:

[ERROR] [com.vmware.loginsight.services.EmbeddedService] [Error when starting service com.vmware.loginsight.analytics.distributed.LogSearchWorkerService]
java.io.IOException: Failed to open bucket index
at com.vmware.loginsight.ingestion.BucketIndex.openReader(BucketIndex.java:574) ~[ingestion-service.jar:?]
at com.vmware.loginsight.ingestion.BucketIndex.getBucketsInfo(BucketIndex.java:459) ~[ingestion-service.jar:?]
at com.vmware.loginsight.ingestion.BucketIndex.getValidBucketInfos(BucketIndex.java:199) ~[ingestion-service.jar:?]
at com.vmware.loginsight.indexing.FSIndexStore.getReloadedBuckets(FSIndexStore.java:45) ~[ingestion-service.jar:?]

Environment

Aria Operations for Logs 8.x

Cause

The issue occurs due to corruption in the Cassandra bucket directory on the node, preventing the loginsight service from reading the bucket index.

This type of corruption typically occurs due to unexpected node shutdowns, often resulting from storage / power outages or similar disruptions.

Resolution

Take a snapshot of all Aria Operations for Logs nodes before proceeding with below resolution.

SSH to the problematic Aria Operations for Logs node
Stop the loginsight service on the node: service loginsight stop
Rename the "/storage/core/loginsight/cidata/store" folder to "/storage/core/loginsight/cidata/store2": mv /storage/core/loginsight/cidata/store /storage/core/loginsight/cidata/store2
Start loginsight service on the node: service loginsight start

Note : This will create new store folder with working buckets directory.

Once the bucket directory has been successfully created, follow the steps below:

Stop the loginsight service again on the affected node: service loginsight stop
Create a shell script "importer.sh" under /tmp and put below two lines in the file

#!/usr/bin/env bash

for bucket in $(ls /storage/core/loginsight/cidata/store | grep -v 'generation\|buckets\|strata_write.lock'); do echo y | /usr/lib/loginsight/application/sbin/bucket-index add $bucket --statuses archived; done

Change the path to tmp directory: cd /tmp
Make importer.sh file executable: chmod +x importer.sh
Run importer.sh script in background mode: nohup ./importer.sh
Now importer is running. You have to monitor process status for this script. Use the command to check if the script is still running: ps aux | grep importer.sh

root@localhost [ ~ ]# ps aux | grep importer
root 1811 0.0 0.0 4124 3032 pts/0 S 15:32 0:00 /bin/bash ./importer.sh
root 1818 0.0 0.0 2452 760 pts/0 S+ 15:32 0:00 grep --color=auto importer

Note: As soon as output of command ps aux | grep importer.sh won't contain line with content importer.sh it means script is completed and you can go ahead and start loginsight service.

Start loginsight service on the node: service loginsight start