"Failed to dispatch hints file" - Cassandra service down - Aria Operations for Logs

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Aria Operations for Logs (formerly vRealize Log Insight) cluster nodes are not responding.
• The Aria Operations for Logs UI is also inaccessible
• The cassandra service crashes with log messages similar to the following seen in /var/log/loginsight/cassandra.log:

INFO [main] 2021-03-31 07:00:23,028 CassandraDaemon.java:556 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
ERROR [HintsDispatcher:1] 2021-03-31 07:00:24,378 HintsDispatchExecutor.java:243 - Failed to dispatch hints file xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-xxxxxxxxxxxxx-1.hints: file is corrupted ({})"

 

 

 

Environment

Aria Operations for Logs 8.x

VMware vRealize Log Insight 8.x

Cause

Cassandra 'hints' file being corrupted which resides under "/usr/lib/loginsight/application/lib/apache-cassandra-*/data/hints".

Resolution

To resolve this issue, follow the below steps to clear out the Cassandra hints files:

Note: Make sure to take a non-memory snapshot of all nodes in the cluster before proceeding.

1. Stop the Log Insight service on all nodes

service loginsight stop

2. Remove the hints files on all nodes

rm -rf /usr/lib/loginsight/application/lib/apache-cassandra-*/data/hints/*

3. Force start the Cassandra service on all nodes

/usr/lib/loginsight/application/sbin/li-cassandra.sh --startnow --force

4. Run this command on all nodes to confirm that the output shows UN for all nodes, from all nodes.

/usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool-no-pass status

Note: The output will look similar to the below

 

5. Force stop the Cassandra service on all nodes

/usr/lib/loginsight/application/sbin/li-cassandra.sh --stopnow --force

6. Start the Log Insight service on all nodes

service loginsight start

7. After a few minutes, the Log Insight UI will be accessible