APM Infrastructure Mainframe traces are not available

Products

• DX SaaS
• DX Operational Observability
• DX Application Performance Management

Issue/Introduction

The Infra agent Mainframe extension has stopped collecting traces, we have tried to restart the APMIA Kafka but didn't help

From Metrics view, we noticed "Kafka Broker Connection Status" is down:

cd  <APMIA>/extensions/kafka-broker-<foldername>/data/kafka_<version>/bin

./kafka-topics.sh --list --bootstrap-server localhost:9092

WARN Could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Environment

DX O2 - APM InfraAgent Mainframe extension 24.*

 

Cause

Zookeeper got corrupted on the APMIA of the mainframe

 

How to verify the condition:

 

cd <APMIA>/extensions/kafka-broker-<foldername>/data/kafka_<version>/bin

 

./zookeeper-server-start.sh <APMIA>/extensions/kafka-broker-<foldername>/data/kafka_<version>/config/zookeeper.properties

 

java.io.IOException: No snapshot found, but there are log entries. Something is broken!

 

 

This inconsistency can occur due to:

1. Improper shutdowns: If ZooKeeper shuts down unexpectedly or crashes, it might not be able to save a snapshot properly.
2. Corruption: Filesystem corruption or other issues might prevent ZooKeeper from reading or writing snapshots correctly.

Resolution

1) Add below property to <APMIA>/extensions/kafka-broker-<foldername>/data/kafka_<version>/config/zookeeper.properties

 

snapshot.trust.empty=true

 

2) Manually restart zookeeper

 

cd <APMIA>/extensions/kafka-broker-<foldername>/data/kafka_<version>/bin

 

./zookeeper-server-start.sh <APMIA>/extensions/kafka-broker-<foldername>/data/kafka_<version>/config/zookeeper.properties

 

You might see some WARN messages but zookeeper process should start, example:

 

 

Verification:

Ensure that the snapshot file exist under zookeeper/version<#>

 

 

Stop the zookeeper process manually, press Ctrl-C

 

3) Restart the Infrastructure Agent 

 

cd <APMIA-HOME>

 

./APMIAgent.sh stop

 

./APMIAgent.sh start

 

 

 

 

RECOMMENDATION: 

Alternatively, upgrade to latest version of  APMIA  25.11 or higher which is bundle with the latest Kafka "without Zookeeper" , so you will not hit this issue in the future

Additional Information

Techdocs APM Agents - Kafka Mainframe extension