Implemented MQ Power Pack monitoring for MQ Queue Manager while the data gathered from 12/05 till 12/08 is missing in APM Investigator.
Below errors are in MQ Power Pack logs
12/05/19 08:30:36 AM EST [INFO] [com.wily.powerpack.websphereMQ.agent.MQMonitor.MQMonitor] Completed processing MQMonitor.properties file.
12/08/19 01:49:16 AM EST [WARN] [IntroscopeAgent.Agent] Enterprise Manager responding too slowly, disconnecting
12/08/19 01:49:16 AM EST [WARN] [IntroscopeAgent.Agent] Heartbeat: Agent, Behavior: Connection Watchdog has been running for excessively long time (120068 ms).
12/08/19 01:49:16 AM EST [WARN] [IntroscopeAgent.Agent] Waited 600000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.server.beans.agent.IAgentBridgeService.recordTimesliceBindingList, v1,.........from thread Agent ServerConnection -- We will keep waiting and don't log further messages until we receive the reply or time out
An overloaded collector caused the performance issue.
Almost 511 Agents report
Over a million transaction traces are reported
Smartstor Duration is way over the recommended limit
Release : 10.7.0
Component : APM Agents
Restarting the Collector EM fixed the issue temporarily.
For a permanent fix, please consider APM Cluster Sizing Performance
Please refer to the below KB for additional information regarding PerfLog.
PM Health Tip - How to improve the performance of APM