Log Insight Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start

Products

VMware Aria Suite

Issue/Introduction

Symptoms:

Unable to start the LogInsight service.
In the /var/log/loginsight/runtime.log file, you see entries similar to:

[2017-06-19 16:30:48.489+0000] ["main"/IP_Address INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Failed to start Cassandra Server: Cassandra failed to start.]
[2017-06-19 16:30:48.489+0000] ["main"/IP_Address INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Exception during start cassandra database]
[2017-06-19 16:30:48.489+0000] ["main"/IP_Address INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Ended start cassandra database at 18396 ms after launch, took 11575 ms]
[2017-06-19 16:30:48.490+0000] ["main"/IP_Address FATAL] [com.vmware.loginsight.daemon.LogInsightDaemon] [Error starting services]
com.vmware.loginsight.daemon.LogInsightDaemon$StartupFailedException: Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start..
at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:757)
at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startCassandraService$16(LogInsightDaemon.java:791)
at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:709)
at com.vmware.loginsight.daemon.LogInsightDaemon.startCassandraService(LogInsightDaemon.java:790)
at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:342)
at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1010)
Caused by: java.lang.Exception: Cassandra failed to start.
at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:746)
In the /storage/var/loginsight/cassandra.log file, you see entries similar to:

WARN [main] 2017-06-19 16:30:47,765 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
WARN [main] 2017-06-19 16:30:47,772 StartupChecks.java:218 - Directory /storage/core/loginsight/cidata/cassandra/commitlog doesn't exist
ERROR [main] 2017-06-19 16:30:47,802 CassandraDaemon.java:638 - Unable to verify sstable files on disk
java.nio.file.FileSystemException: /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context/la-80366-big-CompressionInfo.db: Input/output error

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Running this command to list the files reported in /storage/var/loginsight/cassandra.log file, you see output similar to:

/storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context # ls -lthr

ls: cannot access la-80366-big-CompressionInfo.db: Input/output error
ls: cannot access la-80366-big-Digest.adler32: Input/output error
ls: cannot access la-80366-big-Statistics.db: Input/output error
ls: cannot access la-80366-big-Summary.db: Input/output error
ls: cannot access la-80366-big-Filter.db: Input/output error
total 12K
-????????? ? ? ? ? ? la-80366-big-Summary.db
-????????? ? ? ? ? ? la-80366-big-Statistics.db
-????????? ? ? ? ? ? la-80366-big-Filter.db
-????????? ? ? ? ? ? la-80366-big-Digest.adler32
-????????? ? ? ? ? ? la-80366-big-CompressionInfo.db
drwxr-xr-x 2 root root 4.0K Mar 6 22:26 backups
-rw-r--r-- 1 root root 94 May 28 18:36 la-80366-big-TOC.txt
-rw-r--r-- 1 root root 29 May 28 18:36 la-80366-big-Index.db

Note: The corrupted files may vary for different environments.

Environment

VMware vRealize Log Insight 4.3.x
VMware vRealize Log Insight 8.x
VMware vRealize Log Insight 8.10.x
VMware vRealize Log Insight 8.12.x
VMware vRealize Log Insight 8.14.x
Aria Operations for Logs 8.16.x
Aria Operations for Logs 8.18.x

Cause

This issue occurs due to file system corruption in vRealize Log Insight.

Resolution

To resolve this issue, recreate the corrupted files/directoy in vRealize Log Insight.

Run this command to stop the log insight services:

service loginsight stop
Determine which files/directory is corrupt from the /storage/var/loginsight/cassandra.log file.

For example:

In this case, the corruption is in the vimevent_context directory which is available at /storage/core/loginsight/cidata/cassandra/data/logdb/ folder.
Move the vimevent_context from the location /storage/core/loginsight/cidata/cassandra/data/logdb to a new directory.

Note: If you are not confident using mv command, use copy command to take the backup and then remove the desired directory.

md /storage/core/loginsight/corrupted_files
mv /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context /storage/core/loginsight/corrupted_files
Create a new directory with the same name vimevent_context in the location /storage/core/loginsight/cidata/cassandra/data/logdb.

md /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context
Run this command to start the log insight service:

service loginsight start

Note: This starts the cassandra service first and then the logInsight daemon.

To fix the File system corruption, perform these steps:

Run this command to stop the vRealize Log Insight Service:

service loginsight stop
Run this command to see the mount files:

mount

Note: The output will show /dev/* and mount points.
Run this command to unmount the /storage/core:

umount /storage/core
Run this command to check/repair the filesystem:

/sbin/fsck device

Notes:

Replace device with actual device name.
It may take hours for a bigger disk, plan the downtime accordingly.

Additional Information

Didi two changes:
1. Added 'service loginsight stop' command in the first part of the resolution.
2. Added the latest environment (Aria Operations for Logs 8.18.3)