Aria Operations for Logs UI is inaccessible

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Users are unable to access the Aria Operations for Logs user interface on a primary or worker node.
• The Cassandra service is crashing on that node, and the following errors are observed in the /storage/core/loginsight/var/runtime.log:

[ERROR] [com.vmware.loginsight.services.EmbeddedService] [Error when starting service com.vmware.loginsight.analytics.distributed.LogSearchWorkerService]
java.io.IOException: Failed to open bucket index
        at com.vmware.loginsight.ingestion.BucketIndex.openReader(BucketIndex.java:574) ~[ingestion-service.jar:?]
        at com.vmware.loginsight.ingestion.BucketIndex.getBucketsInfo(BucketIndex.java:459) ~[ingestion-service.jar:?]
        at com.vmware.loginsight.ingestion.BucketIndex.getValidBucketInfos(BucketIndex.java:199) ~[ingestion-service.jar:?]
        at com.vmware.loginsight.indexing.FSIndexStore.getReloadedBuckets(FSIndexStore.java:45) ~[ingestion-service.jar:?]

Environment

Aria Operations for Logs 8.x

Cause

The issue occurs due to corruption in the Cassandra bucket directory on the node, preventing the loginsight service from reading the bucket index.

This type of corruption typically occurs due to unexpected node shutdowns, often resulting from storage / power outages or similar disruptions.

Resolution

Take a snapshot of all Aria Operations for Logs nodes before proceeding with below resolution:

• SSH to the problematic Aria Operations for Logs node.
• Stop the loginsight service on the node:
  
  service loginsight stop
  
  
• Rename the /storage/core/loginsight/cidata/store directory to /storage/core/loginsight/cidata/store2:
  
  mv /storage/core/loginsight/cidata/store /storage/core/loginsight/cidata/store2
  
  
• Start loginsight service on the node:
  
  service loginsight start

Note: This will create new store folder with working buckets directory.

Once the bucket directory has been successfully created, follow the steps below:

• Stop the loginsight service again on the affected node:
  
  service loginsight stop
  
  
• Create a shell script importer.sh under /tmp:

cd /tmp
touch installer.sh
vi installer.sh

• Copy/paste the following into the new installer.sh file:

for bucket in $(ls /storage/core/loginsight/cidata/store | grep -v 'generation\|buckets\|strata_write.lock'); do echo y | /usr/lib/loginsight/application/sbin/bucket-index add $bucket --statuses archived; done

• Change the path to the /tmp directory:
  
  cd /tmp
  
  
• Make importer.sh file executable:
  
  chmod +x importer.sh
  
  
• Run importer.sh script in background mode:
  
  nohup ./importer.sh
  
  
• Now importer is running. You have to monitor process status for this script. Use the following command to check if the script is still running:

ps aux | grep importer.sh

root@localhost [ ~ ]# ps aux | grep importer
root 1811 0.0 0.0 4124 3032 pts/0 S 15:32 0:00 /bin/bash ./importer.sh
root 1818 0.0 0.0 2452 760 pts/0 S+ 15:32 0:00 grep --color=auto importer

Note:  Alternately, you may use the following to get an updated status every two seconds:

watch "ps aux | grep importer.sh"

As soon as the output of the command ps aux | grep importer.sh no longer contains the line with importer.sh it means script is completed and you can go ahead and start loginsight service.

• Start loginsight service on the node:
  
  service loginsight start

• Now try logging into the affected node. If normal operations return, remember to delete all snapshots for each node in the Aria Operations for Logs cluster.