AIOps - kafka crashloop with cluster.id mismatch in meta.properties.

Products

• DX Operational Intelligence
• DX Operational Intelligence
• DX Application Performance Management

Issue/Introduction

The Kafka was not getting started and reports error “Cluster ID doesn’t match stored clusterId in meta.properties”

Environment

AIOps, Data Platform 21.x,

Cause

If the data of the zookeeper is corrupted or deleted and  the zookeeper restarted, there is a chance of a new cluster.id formed, as Kafka is holding the previous cluster.Id, There is a mismatch and crash loop occurs. 

Resolution

1. Get the new Cluster ID from the logs

Example:

[2022-12-06 16:24:25,301] INFO Cluster ID = <token#> (kafka.server.KafkaServer)

2.Find the meta.properties file in the NFS Kafka folder (this file is in <NFS DXI folder>/jarvis/kafka)

Example:

cd /nfs/ca/dxi/jarvis/kafka

find . | grep meta.properties
./kafka-3/meta.properties
./kafka-1/meta.properties
./kafka-2/meta.properties

3. Update the Cluster ID in the meta.properties :

Copy the mismatched cluster.id from the logs and update cluster.id with that value in meta.properties 

Example:

#Wed May 26 
broker.id=0
version=0
cluster.id=<token#>

4. Restart the kafka pod. -- it should able to connect to zookeeper.

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html