Kafka Agent is dropping JMX Metrics
Article ID: 376448
Updated On:
Products
Issue/Introduction
The APMIA Kafka Agent is monitoring 4 Kafka Brokers. After a short period of time, we have observed that certain JMX metrics stop reporting. We have enabled debug logging on the agent, and we are seeing messages such as the following:
[WARN] [IntroscopeAgent.JMXClient] The clamp limit of 50000 messages has been reached in the asynchronous logging queue. No more new log messages will be added.
java.lang.Throwable: Clamp limit reached
at com.wily.util.feedback.AsynchFeedbackChannel$AsyncLoggingBehavior.log(AsynchFeedbackChannel.java:448)
at com.wily.util.feedback.AsynchFeedbackChannel.doLog(AsynchFeedbackChannel.java:282)
at com.wily.util.feedback.ABufferedApplicationFeedbackChannel.logbottleneck(ABufferedApplicationFeedbackChannel.java:152)
at com.wily.util.feedback.AnyBackendFeedbackChannel.log(AnyBackendFeedbackChannel.java:543)
at com.wily.util.feedback.AnyBackendFeedbackChannel.log(AnyBackendFeedbackChannel.java:536)
at com.wily.util.feedback.AnyBackendFeedbackChannel.debug(AnyBackendFeedbackChannel.java:501)
at com.wily.util.feedback.DelegatingFeedbackChannel.debug(DelegatingFeedbackChannel.java:58)
at com.wily.introscope.agent.trace.intelligent.Logger$LoggingHandler.logDebugMessage(Logger.java:200)
at com.ca.apm.agent.jmxclient.api.MixedConfigProvider.loadMBeans(MixedConfigProvider.java:163)
at com.ca.apm.agent.jmxclient.api.MixedConfigProvider.getConfig(MixedConfigProvider.java:75)
at com.ca.apm.agent.jmxclient.api.BeanConfiguration.createSubscribersMap(BeanConfiguration.java:299)
at com.ca.apm.agent.jmxclient.api.BeanConfiguration.createJmxBeanPollers(BeanConfiguration.java:280)
at com.ca.apm.agent.jmxclient.api.BeanConfiguration.access$300(BeanConfiguration.java:74)
at com.ca.apm.agent.jmxclient.api.BeanConfiguration$1.run(BeanConfiguration.java:206)
at com.ca.apm.agent.jmxclient.api.BeanConfiguration$2.run(BeanConfiguration.java:254)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Shortly thereafter, we also start seeing messages as follows:
[DEBUG] [IntroscopeAgent.JMXClient] Stopping polling for jmx bean kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount
After that, the field is no longer displayed within the metric view.
We are using KafkaExtension version 24.3.2.15
Resolution
We determined the root cause of some of our metric reporting issues. We suspect that the issues we are seeing relate to the Agent attempting to pull metrics before the Broker is fully initialized and ready to take requests. We did a time delay in the agent settings on startup and that resolved it.
Feedback
