VMware Aria Operations Cluster fails to come online and stuck on "Waiting for Analytics"

Products

• VMware Aria Suite

Issue/Introduction

The purpose of this article is to provide steps to remove Cassandra Commit log corruption

• Analytics service fails to come online logs shows similar errors to :

 ERROR [WrapperSimpleAppMain] CommitLogReplayer.java:389 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 41587836 of commit log /storage/db/vcops/cassandra/commit-log/CommitLog-6-1673197048694.log, with invalid CRC. The end of segment marker should be zero.</pre>

 

 

Environment

VMware Aria Operations (formerly vRealize Operations) 8.6 and earlier versions

Resolution

Important: Take snapshots of each of the VMware Aria Operations nodes before applying any of the steps below How to take a Snapshot of VMware Aria Operations.

 

1. Bring Aria Operations cluster offline through the admin UI

2. Open ssh session as root to the affected node

3. Delete the affected commit log files /storage/db/vcops/cassandra/commit-log/CommitLog-* 

rm -rf /storage/db/vcops/cassandra/commit-log/CommitLog-* 

4. Bring cluster back online from the admin UI

Additional Information

Impact/Risks:

Cluster will not go online