Log Insight Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Unable to start the LogInsight service.
• In the /var/log/loginsight/runtime.log file, you see entries similar to:
  
  [2017-06-19 16:30:48.489+0000] ["main"/172.16.146.240 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Failed to start Cassandra Server: Cassandra failed to start.]
  [2017-06-19 16:30:48.489+0000] ["main"/172.16.146.240 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Exception during start cassandra database]
  [2017-06-19 16:30:48.489+0000] ["main"/172.16.146.240 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Ended start cassandra database at 18396 ms after launch, took 11575 ms]
  [2017-06-19 16:30:48.490+0000] ["main"/172.16.146.240 FATAL] [com.vmware.loginsight.daemon.LogInsightDaemon] [Error starting services]
  com.vmware.loginsight.daemon.LogInsightDaemon$StartupFailedException: Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start..
  at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:757)
  at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startCassandraService$16(LogInsightDaemon.java:791)
  at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:709)
  at com.vmware.loginsight.daemon.LogInsightDaemon.startCassandraService(LogInsightDaemon.java:790)
  at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:342)
  at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1010)
  Caused by: java.lang.Exception: Cassandra failed to start.
  at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:746)
  
  
• In the /storage/var/loginsight/cassandra.log file, you see entries similar to:
  
  WARN [main] 2017-06-19 16:30:47,765 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
  WARN [main] 2017-06-19 16:30:47,772 StartupChecks.java:218 - Directory /storage/core/loginsight/cidata/cassandra/commitlog doesn't exist
  ERROR [main] 2017-06-19 16:30:47,802 CassandraDaemon.java:638 - Unable to verify sstable files on disk
  java.nio.file.FileSystemException: /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context/la-80366-big-CompressionInfo.db: Input/output error
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
  
  
• Running this command to list the files reported in /storage/var/loginsight/cassandra.log file, you see output similar to:
  
  /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context # ls -lthr
  
  ls: cannot access la-80366-big-CompressionInfo.db: Input/output error
  ls: cannot access la-80366-big-Digest.adler32: Input/output error
  ls: cannot access la-80366-big-Statistics.db: Input/output error
  ls: cannot access la-80366-big-Summary.db: Input/output error
  ls: cannot access la-80366-big-Filter.db: Input/output error
  total 12K
  -????????? ? ? ? ? ? la-80366-big-Summary.db
  -????????? ? ? ? ? ? la-80366-big-Statistics.db
  -????????? ? ? ? ? ? la-80366-big-Filter.db
  -????????? ? ? ? ? ? la-80366-big-Digest.adler32
  -????????? ? ? ? ? ? la-80366-big-CompressionInfo.db
  drwxr-xr-x 2 root root 4.0K Mar 6 22:26 backups
  -rw-r--r-- 1 root root 94 May 28 18:36 la-80366-big-TOC.txt
  -rw-r--r-- 1 root root 29 May 28 18:36 la-80366-big-Index.db
  
  Note: The corrupted files may vary for different environments.
  

Environment

VMware vRealize Log Insight 4.3.x

Cause

This issue occurs due to file system corruption in vRealize Log Insight.

Resolution

To resolve this issue, recreate the corrupted files/directoy in vRealize Log Insight.

1. Determine which files/directory is corrupt from the /storage/var/loginsight/cassandra.log file.
   
   For example:
   
   In this case, the corruption is in the vimevent_context directory which is available at /storage/core/loginsight/cidata/cassandra/data/logdb/ folder.
   
   
   
2. Move the vimevent_context from the location /storage/core/loginsight/cidata/cassandra/data/logdb to a new directory.
   
   Note: If you are not confident using mv command, use copy command to take the backup and then remove the desired directory.
   
   md /storage/core/loginsight/corrupted_files
   mv /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context /storage/core/loginsight/corrupted_files
   
   
   
3. Create a new directory with the same name vimevent_context in the location /storage/core/loginsight/cidata/cassandra/data/logdb.
   
   md /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context
   
   
4. Run this command to start the logInsight service:
   
   service logInsight start
   
   Note: This starts the cassandra service first and then the logInsight daemon.

To fix the File system corruption, perform these steps:

1. Run this command to stop the vRealize Log Insight Service:
   
   service loginsight stop
   
   
2. Run this command to see the mount files:
   
   mount
   
   Note: The output will show /dev/* and mount points.
   
   
3. Run this command to unmount the /storage/core:
   
   umount /storage/core
   
   
4. Run this command to check/repair the filesystem:
   
   /sbin/fsck device
   
   Notes:

• Replace device with actual device name.
• It may take hours for a bigger disk, plan the downtime accordingly.