How to troubleshoot Kafka streaming metric export from Data Aggregator

Products

• CA Performance Management
• Network Observability

Issue/Introduction

Step-by-step debugging of Kafka streaming metric export from Data Aggregator 

Environment

Release : All DX Netops supported versions

Resolution

Here’s step-by-step debugging of Kafka streaming metric export from DA:

1. Check the settings in DA’s $KARAF_HOME/etc/kafkaexport.producer.cfg file are correct.
2. Check for ‘ProducerStatisticsMonitor’ in DA’s $KARAF_HOME/data/log/karaf.log file. These are emitted every five minutes by default.
3. If ProducerStatisticsMonitor shows all messages as sent, go to step 6.
4. If ProducerStatisticsMonitor shows that messages are being dropped, look in DA’s $KARAF_HOME/data/log/KafkaLog.log file for errors/hints to the problem.
5. If ProducerStatisticsMonitor logs are not seen, check whether export configuration has been set up and applied to devices:
   1. In a browser, open http(s)://<DAHOST>:<DAPORT>/debug
   2. Click on Available Spring Containers (by bundle)
   3. Provide PC admin credentials if prompted
   4. Click on com.ca.im.data-manager.core.aggregator.loader.integrator bundle link
   5. Click on exportProfileCache link
   6. Verify that there is an ExportProfileConfig defined and it has the expected exportedMetricFamilyQNames.
   7. Verify that the exportedDeviceCout (sic) is non-zero.
   8. If there is no ExportProfileConfig or there are no exported devices associated, check DA’s karaf.log file or OIConnector logs for failures in creating the config or associating it with collections.
6. Verify that the messages are truly getting to Kafka topic. On the Kafka broker (based on defaults):
/opt/CA/kafka/kafkadist/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic metric-export
7. If no data is flowing, then check kafka/zookeeper logs for potential problems.