Various EM problems start occurring with EMs
Closer investigation finds
1. The Perflog shows a gap of 10 or 20 minutes.
2. Messages in the EM Log such as
11/30/16 11:05:03.584 PM EST [ERROR] [PO:main Mailman 2] [Manager] AgentThresholdDeliveryService: unable to get trigger service for agent: SuperDomain|abc04g|WebSphere|corT2-abc04g/monitorServer2 [state=Connected, ipAddress=10.0.0.1, socketType=default, okToDisconnect=true, okToUnmount=true, okToAutoUnmount=true, supportsShutoff=true, shutoff=false, supportsTransactionTracing=true, supportedTransactionTracingFilterTypes=(0,1,2,3,4,5,6,7, dynamicInstrumentationFlags=0]
com.wily.isengard.messageprimitives.TimeoutConnectionException: Service call to host {Unknown} timed out after 2000 ms com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.agent.beans.autotracing.IAutoTracingTriggerService.clearAllAutoTracingTriggers, v1, []} threadname PO:main Mailman 2
at com.wily.isengard.messageprimitives.service.MessageServiceClient.blockOnResponse(MessageServiceClient.java:282)
at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:163)
at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:356)
at com.sun.proxy.$Proxy163.clearAllAutoTracingTriggers(Unknown Source)
at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl.agentAdded(AgentThresholdDeliveryServiceImpl.java:218)
at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl$AutoTraceTriggerQuery.dataAdded(AgentThresholdDeliveryServiceImpl.java:632)
at com.wily.isengard.ongoingquery.AbstractQueryServiceManager$NotifyAd
EM hangs due to large amount of Agents connecting at same time. Various temporary connection and performance issues may manifest this issue.
This is fixed in APM 10.3. Earlier releases have hotfixes available --10.1 HF#25, 10.2 HF#24