Data Collector ActiveMQ Pending messages increasing in DX NetOps Performance Management

book

Article ID: 211581

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

I am seeing the Pending Message on DC kept going up, even though DC/DA are connected fine, system status showing all green. The few random graphs that I checked are showing updated data.

It seems like the Enqueue is always larger than the Dequeue number.

When debugging JmsBrokerHealthAnalyser messages on the Data Collector we can monitor the ActiveMQ cache processing. It shows us whether or not it's queue's are being processed or not.

The debug is enabled by adding these lines to the org.ops4j.pax.logging.cfg file on the Data Collector (DC) in (default path) /opt/IMDataCollector/apache-karaf-<version>/etc

#
# Monitor DC AMQ cache burndown
# Writes to com.ca.im.common.core.jms.log on DC
#
log4j.logger.com.ca.im.core.jms.health.JmsBrokerHealthAnalyser=DEBUG,sift
log4j.additivity.com.ca.im.core.jms.health.JmsBrokerHealthAnalyser=false

It creates logs named com.ca.im.common.core.jms.log in the (default path) /opt/IMDataCollector/apache-karaf-<version>/data/log directory. Sample messages look like this.

2021-03-24 11:54:31,073 | DEBUG | pool-13-thread-1 | JmsBrokerHealthAnalyser          | s.health.JmsBrokerHealthAnalyser  121 |  | JMS Health: received statistics: temp=0, size=8
2021-03-24 11:54:31,085 | DEBUG | pool-13-thread-1 | JmsBrokerHealthAnalyser          | s.health.JmsBrokerHealthAnalyser  149 |  | JMS Health Statistics => Memory: 9.10KB/678.13MB, Disk: 0/20.00GB, Pending: 8 msgs, Enqueue: -1 msg/sec, Dequeue: -1 msg/sec, Delay: -1 secs, Dropped: 0 msgs

When watching the messages in those logs, the Pending count continually rises, then drops, then rises again. Is this normal?

Environment

All supported DX NetOps Performance Management releases

Resolution

If we see the pending counts for each hour at the end of the hour (generally around 55 minutes into the hour), and just after the start of the hour (generally 5 minutes after the start of the hour), is that where a change to the counts is seen where they drop before rising again?

If that is what we observe it is expected behavior. The messages held in the AMQ Pending queues on the DC or related to roll up processing. When it runs just before the end of the hour, and/or just after the start of the hour, the Pending queue counts will go down. When that completes they'll begin to grow again until the next roll up process cycles start again.

It is only if we do not see the Pending messages being addressed and lowering in a cyclical manner that we should be concerned. It would generally mean there is a problem with roll up processing and/or execution.