Failed to start Cassandra Server: Cassandra failed to start
search cancel

Failed to start Cassandra Server: Cassandra failed to start

book

Article ID: 426953

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Unable to start the cassandra service.
In the /storage/var/loginsight/cassandra.log file, you see error entries similar to:
ERROR [SSTableBatchOpen:2] ####-##-##:##:##,### DefaultFSErrorHandler.java:### - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /storage/core/loginsight/cidata/cassandra/data/system/size_estimates-###################8e86/nb-####-big-Statistics.db
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:483) ~[apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:381) ~[apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader$2.run(SSTableReader.java:551) [apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:124) [apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) [apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) [apache-cassandra-4.1.9.jar:4.1.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.58.Final.jar:4.1.58.Final]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.io.EOFException: EOF after 3971 bytes out of 4545
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:70) ~[apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:62) ~[apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:193) ~[apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:127) ~[apache-cassandra-4.1.9.jar:4.1.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:479) ~[apache-cassandra-4.1.9.jar:4.1.9]
        ... 9 more

Environment

Aria Operations for Logs 8.18.x

Resolution

To resolve this issue, move the corrupted file to different folder:

1. Stop the log insight service: 
systemctl stop loginsight
 
2. Determine which directory contains the corrupt files from the ERROR message in the /storage/var/loginsight/cassandra.log file. For example:
- Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /storage/core/loginsight/cidata/cassandra/data/system/size_estimates-###################8e86/nb-####-big-Statistics.db
 
3. Move the corrupted directory from the current location to a new location:
mkdir /storage/core/loginsight/corrupted_files
Note: Replace [location] with the location identified in Step 2. Example:
mv /storage/core/loginsight/cidata/cassandra/data/system/size_estimates-###################8e86/nb-####-big-Statistics.db /storage/core/loginsight/corrupted_files
 
4. Start the log insight service:
systemctl start loginsight

Additional Information

Refer: Log Insight Daemon startup failed: Failed to start Cassandra Server: Cassandra failed to start

Powered by Wolken Software