Kafka and data-insights-dbloader pods are failing to restart in Watchtower with default password

Products

• WatchTower
• WatchTower Platform

Issue/Introduction

Kafka and data-insights-dbloader pods are failing to restart in Watchtower v1.3.2 with default password.   WT Diagnostics Report provided. 

Environment

WatchTower v1.3.x

Cause

In review of the Diagnostics Report: 

Kafka fails to start because the JKS keystore in secret datastream-tls-secret cannot be unlocked with the hardcoded password "changeme".
The dbloader crash is a downstream consequence — it cannot resolve the Kafka bootstrap server at startup.

KAFKA_SSL_KEYSTORE_PASSWORD - changeme
KAFKA_SSL_TRUSTSTORE_PASSWORD - changeme

Kafka cannot unlock the JKS keystore in the datastream-tls-secret because the configured password KAFKA_SSL_KEYSTORE_PASSWORD is set to the default "changeme", which does not match the actual password used when the certificates were generated.

Resolution

 

1. Identify the correct password: Retrieve the actual password used when the broker.ks keystore was generated (check the CI/CD secrets or certificate provisioning pipeline).
2. Update Helm values: Update both KAFKA_SSL_KEYSTORE_PASSWORD and KAFKA_SSL_TRUSTSTORE_PASSWORD in your Helm values.yaml with the correct password.
3. Force a clean restart: Delete the Kafka pod to apply the changes: kubectl delete pod kafka-0 -n itid-ikp-watchtower-uk-dev
4. Verify Recovery: Once the kafka-0 pod is healthy and its service endpoint is ready, the data-insights-dbloader pod will self-recover on its next restart.