Log Ingestion Fails After Upgrade to 25.1 Due to Kafka Configuration Overwrite
search cancel

Log Ingestion Fails After Upgrade to 25.1 Due to Kafka Configuration Overwrite

book

Article ID: 440233

calendar_today

Updated On:

Products

DX Operational Observability

Issue/Introduction

After upgrading to version 25.1, log ingestion stops working even though all pods appear to be in a 'Running' state. This issue is typically a regression from version 24.1 where ingestion was functioning correctly.

Environment

  • DX Operational Observability / Log for Triage
  • Version: 25.1 Upgrade

Cause

During the upgrade to 25.1, the 0001_kafka.conf configuration file for the logs-logparser pod is incorrectly overwritten. The bootstrap_servers entry is changed to ${KAFKA_URLS:kafka:9092}.

Because the environment variable KAFKA_URLS is not defined within the pod, the configuration falls back to the default value kafka:9092. In most environments, this default endpoint is invalid, causing the log parser to fail its connection to the Kafka cluster.

Additionally, a known behavior in 25.1 causes the system to repeatedly append "generated" data blocks to 0001_kafka.conf upon each pod restart, leading to file bloat, though this is secondary to the connection failure.

Resolution

Correct to DX Operational Observability cluster

  1. Identify the affected pod: Locate the logs-logparser pod in your cluster.
  2. Edit the configuration: Access the pod and locate the file: 0001_kafka.conf.
  3. Modify the Bootstrap URL: Locate the bootstrap_servers parameter within the kafka input block.
  4. Update the value: Change the entry from: bootstrap_servers => "${KAFKA_URLS:kafka:9092}" to: bootstrap_servers => "${KAFKA_URLS_DEFAULT}"
  5. Restart the service: Restart the logs-logparser pod to reload the corrected configuration.

Verification

  • Monitor the logs-logparser logs to ensure it is no longer attempting to connect to kafka:9092.
  • Confirm that logs are once again appearing in the DX OI Dashboards.

Additional Information

If the logs-logparser pod is in a restart loop, inspect the 0001_kafka.conf for excessive duplicate entries. While the duplication is considered benign regarding functionality, it can indicate high restart frequency which should be investigated as a separate issue.

Powered by Wolken Software