EM is very slow to come up or hangs after a restart or network glitch. Many components do not work correctly.

Products

• APP PERF MANAGEMENT
• CA Application Performance Management Agent (APM / Wily / Introscope)
• CUSTOMER EXPERIENCE MANAGER
• INTROSCOPE

Issue/Introduction

    Various EM problems start occurring with EMs

•    EM fail to startup
•    EMs get hung or respond slowly for a period of time. In some cases, a restart may be needed
•    After a network 'glitch', the Enterprise Manager becomes unresponsive and often does not recover and needs to be restarted. 
•    The MOM keeps disconnecting from the collectors or the collectors stop communicating with the MOM.
•    High spikes in harvest durations.

 

Closer investigation finds 

        1. The Perflog shows a gap of 10 or 20 minutes.

        2. Messages in the EM Log such as 

11/30/16 11:05:03.584 PM EST [ERROR] [PO:main Mailman 2] [Manager] AgentThresholdDeliveryService: unable to get trigger service for agent: SuperDomain|abc04g|WebSphere|corT2-abc04g/monitorServer2 [state=Connected, ipAddress=10.0.0.1, socketType=default, okToDisconnect=true, okToUnmount=true, okToAutoUnmount=true, supportsShutoff=true, shutoff=false, supportsTransactionTracing=true, supportedTransactionTracingFilterTypes=(0,1,2,3,4,5,6,7, dynamicInstrumentationFlags=0] 

com.wily.isengard.messageprimitives.TimeoutConnectionException: Service call to host {Unknown} timed out after 2000 ms com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.agent.beans.autotracing.IAutoTracingTriggerService.clearAllAutoTracingTriggers, v1, []} threadname PO:main Mailman 2 

at com.wily.isengard.messageprimitives.service.MessageServiceClient.blockOnResponse(MessageServiceClient.java:282) 

at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:163) 

at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:356) 

at com.sun.proxy.$Proxy163.clearAllAutoTracingTriggers(Unknown Source) 

at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl.agentAdded(AgentThresholdDeliveryServiceImpl.java:218) 

at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl$AutoTraceTriggerQuery.dataAdded(AgentThresholdDeliveryServiceImpl.java:632) 

at com.wily.isengard.ongoingquery.AbstractQueryServiceManager$NotifyAd 

 

Cause

EM hangs due to large amount of Agents connecting at same time. Various temporary connection and performance issues may manifest this issue.

 

Environment

EMs running APM 10.1 and 10.2 . This is fixed in APM 10.3

Resolution

 This is fixed in APM 10.3. Earlier releases have hotfixes available --10.1 HF#25, 10.2 HF#24