Log Insight service keeps crashing with error: "StartupFailedException: Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start."

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

• Log Insight service keeps crashing.
   
• In runtime.log (/var/log/loginsight), you may find similar error:
  
  

  [INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Shutdown complete in 0 ms]
  [####-##-## ##:##:##.###+0000] ["ActiveMQ ShutdownHook"/##.###.#.### INFO] [org.apache.activemq.broker.BrokerService] [Apache ActiveMQ 5.11.1 
  [ERROR] [com.vmware.loginsight.daemon.DaemonCommandsHandler] [Error continuing upgrade.]
  StartupException(description:com.vmware.loginsight.daemon.LogInsightDaemon$StartupFailedException: Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start..)
  at com.vmware.loginsight.daemon.DaemonCommandsHandler.waitUntilStarted(DaemonCommandsHandler.java)
  at com.vmware.loginsight.daemon.DaemonCommandsHandler$1.run(DaemonCommandsHandler.java)
  at java.lang.Thread.run(Unknown Source)

• In cassandra.log (/var/log/loginsight), you may find simialr error:

  INFO [main] ####-##-## ##:##:##,### ColumnFamilyStore.java:### - Initializing logdb.vimevent_context
  ERROR [SSTableBatchOpen:1] ####-##-## ##:##:##,### FileUtils.java:### - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
  org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context-###################/la-####-big-CompressionInfo.db
  at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:###) ~[apache-cassandra-####.###]

  Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

Aria Operations for Logs 8.18.x

Resolution

To resolve the issue follow the below steps:

1. Take snapshot of the nodes
2. Stop log insight service. 

   systemctl stop loginsight

3. Determine the corrupted folder from cassandra.log and move the folder to a backup location.
   For example:

   mkdir /storage/core/loginsight/backup
   mv /storage/core/loginsight/cidata/cassandra/data/system/compaction_history-######################### /storage/core/loginsight/backup

    (the location is from the cassandra.log)
4. Create the folder again with the same name at the same location as per the log.
   For example:

   mkdir /storage/core/loginsight/cidata/cassandra/data/system/compaction_history-#########################)

5. Start the log insight service daemon.

   systemctl start loginsight

6. Login to the application using web browser and check Configuration --> cluster for nodes status.