Seeing gaps in metric intermittently for multiple agents in Prod tenant.

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

We are seeing this with an agent which is NOT using cloudproxy. 

 

 [INFO] [IntroscopeAgent.ReportingQueue] Reporting Queue has reached max metric limit 14979

[WARN] [IntroscopeAgent.InfrastructureAgent] Waited 30000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.server.beans.agent.IAgentBridgeService.recordTimesliceBindingList, v1, [com.wily.introscope.spec.metric.CompressingAgentMetricDataSet2@<num>], source: Agent_<num>.client_main:264, remoteHost: {Unknown}} from address Agent_<number>.client_main:264 to service address Server.WatchedAgentPO:68335 from thread Agent ServerConnection -- We will not wait any longer

[ERROR] [I.IsengardServerConnectionManager] Agent reporter thread caught unexpected exception.

[WARN] [I.IsengardServerConnectionManager] Lost contact with the Introscope Enterprise Manager at apmgw.dxi-na1.saas.broadcom.com:443.

[WARN] [IntroscopeAgent.InfrastructureAgent] Heartbeat: InfrastructureAgent, Behavior: Connection Watchdog has been running for excessively long time (119808 ms).

Resolution

A review was given. The APM agents were actually using cloud proxy to report to the prod tenant like with the other DX APM SaaS tenants. There were issues with the proxy as well.

It might be worth reverting back to the older proxy settings to see if it helps make the agent connections to  prod tenant more stable. It is best to monitor the situation along with our devops monitoring on the DX APM SaaS tenants and capture logs, preferably in DEBUG mode, from the cloud proxy and the problem agents if the data gap issue occurs again.