Log Insight Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start
search cancel

Log Insight Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start

book

Article ID: 337464

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
  • Unable to start the LogInsight service.
  • In the /var/log/loginsight/runtime.log file, you see entries similar to:

    [2017-06-19 16:30:48.489+0000] ["main"/172.16.146.240 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Failed to start Cassandra Server: Cassandra failed to start.]
    [2017-06-19 16:30:48.489+0000] ["main"/172.16.146.240 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon]
    [Exception during start cassandra database]
    [2017-06-19 16:30:48.489+0000] ["main"/172.16.146.240 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Ended start cassandra database at 18396 ms after launch, took 11575 ms]
    [2017-06-19 16:30:48.490+0000] ["main"/172.16.146.240 FATAL] [com.vmware.loginsight.daemon.LogInsightDaemon] [Error starting services]
    com.vmware.loginsight.daemon.
    LogInsightDaemon$StartupFailedException: Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start..
    at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:757)
    at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startCassandraService$16(LogInsightDaemon.java:791)
    at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:709)
    at com.vmware.loginsight.daemon.LogInsightDaemon.startCassandraService(LogInsightDaemon.java:790)
    at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:342)
    at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1010)
    Caused by: java.lang.Exception: Cassandra failed to start.
    at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:746)

  • In the /storage/var/loginsight/cassandra.log file, you see entries similar to:

    WARN [main] 2017-06-19 16:30:47,765 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
    WARN [main] 2017-06-19 16:30:47,772 StartupChecks.java:218 - Directory /storage/core/loginsight/cidata/cassandra/commitlog doesn't exist
    ERROR [main] 2017-06-19 16:30:47,802 CassandraDaemon.java:638 - Unable to verify sstable files on disk
    java.nio.file.FileSystemException:
    /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context/la-80366-big-CompressionInfo.db: Input/output error

    Note
    : The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

  • Running this command to list the files reported in /storage/var/loginsight/cassandra.log file, you see output similar to:

    /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context # ls -lthr

    ls: cannot access la-80366-big-CompressionInfo.db: Input/output error
    ls: cannot access la-80366-big-Digest.adler32: Input/output error
    ls: cannot access la-80366-big-Statistics.db: Input/output error
    ls: cannot access la-80366-big-Summary.db: Input/output error
    ls: cannot access la-80366-big-Filter.db: Input/output error
    total 12K
    -????????? ? ? ? ? ? la-80366-big-Summary.db
    -????????? ? ? ? ? ? la-80366-big-Statistics.db
    -????????? ? ? ? ? ? la-80366-big-Filter.db
    -????????? ? ? ? ? ? la-80366-big-Digest.adler32
    -????????? ? ? ? ? ? la-80366-big-CompressionInfo.db
    drwxr-xr-x 2 root root 4.0K Mar 6 22:26 backups
    -rw-r--r-- 1 root root 94 May 28 18:36 la-80366-big-TOC.txt
    -rw-r--r-- 1 root root 29 May 28 18:36 la-80366-big-Index.db


    Note: The corrupted files may vary for different environments.


Environment

VMware vRealize Log Insight 4.3.x

Cause

This issue occurs due to file system corruption in vRealize Log Insight.

Resolution

To resolve this issue, recreate the corrupted files/directoy in vRealize Log Insight.

  1. Determine which files/directory is corrupt from the /storage/var/loginsight/cassandra.log file.

    For example:

    In this case, the corruption is in the vimevent_context directory which is available at /storage/core/loginsight/cidata/cassandra/data/logdb/ folder.


  2. Move the vimevent_context from the location /storage/core/loginsight/cidata/cassandra/data/logdb to a new directory.

    Note: If you are not confident using mv command, use copy command to take the backup and then remove the desired directory.

    md /storage/core/loginsight/corrupted_files
    mv /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context /storage/core/loginsight/corrupted_files



  3. Create a new directory with the same name vimevent_context in the location /storage/core/loginsight/cidata/cassandra/data/logdb.

    md /storage/core/loginsight/cidata/cassandra/data/logdb/vimevent_context

  4. Run this command to start the logInsight service:

    service logInsight start

    Note: This starts the cassandra service first and then the logInsight daemon.

To fix the File system corruption, perform these steps:

  1. Run this command to stop the vRealize Log Insight Service:

    service loginsight stop

  2. Run this command to see the mount files:

    mount

    Note: The output will show /dev/* and mount points.

  3. Run this command to unmount the /storage/core:

    umount /storage/core

  4. Run this command to check/repair the filesystem:

    /sbin/fsck device

    Notes:
    • Replace device with actual device name.
    • It may take hours for a bigger disk, plan the downtime accordingly.