Data Collector ActiveMQ Pending messages increasing in DX NetOps Performance Management
Article ID: 211581
Updated On:
Products
Issue/Introduction
I am seeing the Pending Message on DC kept going up, even though DC/DA are connected fine, system status showing all green. The few random graphs that I checked are showing updated data.
It seems like the Enqueue is always larger than the Dequeue number.
When debugging JmsBrokerHealthAnalyser messages on the Data Collector we can monitor the ActiveMQ cache processing. It shows us whether or not it's queue's are being processed or not.
The debug is enabled by adding these lines to the org.ops4j.pax.logging.cfg file on the Data Collector (DC) in (default path) /opt/IMDataCollector/apache-karaf-<version>/etc
#
# Monitor DC AMQ cache burndown
# Writes to com.ca.im.common.core.jms.log on DC
#
log4j.logger.com.ca.im.core.jms.health.JmsBrokerHealthAnalyser=DEBUG,sift
log4j.additivity.com.ca.im.core.jms.health.JmsBrokerHealthAnalyser=false
It creates logs named com.ca.im.common.core.jms.log in the (default path) /opt/IMDataCollector/apache-karaf-<version>/data/log directory. Sample messages look like this.
2021-03-24 11:54:31,073 | DEBUG | pool-13-thread-1 | JmsBrokerHealthAnalyser | s.health.JmsBrokerHealthAnalyser 121 | | JMS Health: received statistics: temp=0, size=8
2021-03-24 11:54:31,085 | DEBUG | pool-13-thread-1 | JmsBrokerHealthAnalyser | s.health.JmsBrokerHealthAnalyser 149 | | JMS Health Statistics => Memory: 9.10KB/678.13MB, Disk: 0/20.00GB, Pending: 8 msgs, Enqueue: -1 msg/sec, Dequeue: -1 msg/sec, Delay: -1 secs, Dropped: 0 msgs
When watching the messages in those logs, the Pending count continually rises, then drops, then rises again. Is this normal?
Environment
All supported DX NetOps Performance Management releases
Resolution
If we see the pending counts for each hour at the end of the hour (generally around 55 minutes into the hour), and just after the start of the hour (generally 5 minutes after the start of the hour), is that where a change to the counts is seen where they drop before rising again?
If that is what we observe it is expected behavior. The messages held in the AMQ Pending queues on the DC or related to roll up processing. When it runs just before the end of the hour, and/or just after the start of the hour, the Pending queue counts will go down. When that completes they'll begin to grow again until the next roll up process cycles start again.
It is only if we do not see the Pending messages being addressed and lowering in a cyclical manner that we should be concerned. It would generally mean there is a problem with roll up processing and/or execution.
Additional Information
The debug logging method noted in this articles Introduction does NOT apply to DX NetOps Performance Management releases r21.2.2 and newer.
It applies ONLY to r20.2.x and earlier releases.
It does not work in r21.2.2+ releases.
It works only with apache-karaf-2.4.3, not the new apache-karaf-4.2.6.
Steps to enable the debug in the new releases can be found in the Enabling JMS Health Debug logging in NetOps r21.2.2+ releases Knowledge Base article.
Feedback
