Kafka and MinIO pods failing after API Developer Portal upgrade to 5.4

Products

• CA API Developer Portal

Issue/Introduction

After upgrading the CA API Developer Portal to version 5.4 in a Kubernetes environment, the Kafka and MinIO pods fail to start or enter a CrashLoopBackOff state.

Kafka Error Logs: [2026-03-16 09:44:50,005] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
kafka.common.InconsistentBrokerIdException: Configured broker.id 1 doesn't match stored broker.id Some(3) in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
at kafka.server.KafkaServer.getOrGenerateBrokerId(KafkaServer.scala:1024)
at kafka.server.KafkaServer.startup(KafkaServer.scala:245)
at kafka.Kafka$.main(Kafka.scala:113)
at kafka.Kafka.main(Kafka.scala)
[2026-03-16 09:44:50,007] INFO shutting down (kafka.server.KafkaServer)

MinIO Error Logs: ERROR Unable to initialize backend: Disk /opt/data: open /opt/data/.minio.sys/format.json: operation not permitted

Environment

API Developer Portal 5.4 (Kubernetes installation)

Cause

1. Kafka: The broker.id logic in the 5.4 deployment fails to align with the existing meta.properties file stored in the persistent volume (PVC) from the previous version. This occurs when the BROKER_ID_COMMAND does not correctly parse the hostname to match the previously assigned ID.
2. MinIO: The pod lacks the necessary filesystem permissions to access the initialization files in /opt/data. This is typically caused by a mismatch in the podSecurityContext (UID/GID) settings following the upgrade.

Resolution

1. Correct Kafka Broker ID Mapping

Update the Kafka ConfigMap to ensure the Broker ID is derived correctly from the pod hostname by modifying the BROKER_ID_COMMAND.

• Locate the Kafka ConfigMap or the relevant section in your Helm values.yaml file.
• Old value: BROKER_ID_COMMAND: echo $HOSTNAME | rev | cut -d "-" -f 1
• New Value: BROKER_ID_COMMAND: echo $HOSTNAME | sed 's/.*-//' 

. Update MinIO Security Context and Resources

Adjust the security context to grant the MinIO process proper access to the persistent storage and increase memory to prevent Out of Memory (OOM) failures.

• In your MinIO deployment configuration or values.yaml, set the podSecurityContext:
   

  podSecurityContext:  fsGroup: 2000  runAsGroup: 2000  runAsUser: 1000

• Optional: If minio-3 pods show OOM errors, increase the memory limit from 256M to 512M.

3. Reset Analytics Supervisors

After the pods are stable and "Running," you must reset the analytics collection tasks to clear any stuck offsets.

1. Connect to the coordinator pod:
   bash

   kubectl exec -it <coordinator-pod-name> -- /bin/bash

2. Execute the following reset commands:
   bash

   curl -X POST http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics/resetcurl -X POST http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics_hour/reset