DX O2 jarvis-kafka in CrashLoopBackOff with error “Cluster ID doesn’t match stored clusterId in meta.properties”

Products

• DX Operational Observability

Issue/Introduction

The jarvis-kafka-* pod log reports the error “Cluster ID doesn’t match stored clusterId in meta.properties”

Environment

DX O2 2* OnPremise

Cause

Mismatch of clustrer id due to a corruption : zookeeper is corrupted and after a restart a new cluster.id has been created, however Kafka is holding the previous cluster.Id, this mismatch is causing the continuous restart. 

Resolution

1. Get the new Cluster ID from the logs

Example:

INFO Cluster ID = <token#> (kafka.server.KafkaServer)

2.Find the meta.properties file in the NFS Kafka folder (this file is in <NFS DXI folder>/jarvis/kafka)

Example:

cd <NFS-folder>/dxi/jarvis/kafka

find . | grep meta.properties
./kafka-3/meta.properties
./kafka-1/meta.properties
./kafka-2/meta.properties

3. Update the Cluster ID in the meta.properties :

Copy the mismatched cluster.id from the logs and update cluster.id with that value in meta.properties 

Example:

#Wed May 26 
broker.id=0
version=0
cluster.id=<token#>

4. Delete the the jarvis-kafka pod, a new pod will be created and jarvis-kafka should able to connect to the jarvis-zookeeper