EM is very slow to come up or hangs after a restart or network glitch. Many components do not work correctly.

Article Id: 5846
Status: Published
Updated On: 14-02-2018 08:10
Legacy Id: TEC1423119
Products:
CA Application Performance Management Agent (APM / Wily / Introscope) , INTROSCOPE
Issue/Introduction:

    Various EM problems start occurring with EMs

  •    EM fail to startup
  •    EMs get hung or respond slowly for a period of time. In some cases, a restart may be needed
  •    After a network 'glitch', the Enterprise Manager becomes unresponsive and often does not recover and needs to be restarted. 
  •    The MOM keeps disconnecting from the collectors or the collectors stop communicating with the MOM.
  •    High spikes in harvest durations.

 

Closer investigation finds 

        1. The Perflog shows a gap of 10 or 20 minutes.

        2. Messages in the EM Log such as 

11/30/16 11:05:03.584 PM EST [ERROR] [PO:main Mailman 2] [Manager] AgentThresholdDeliveryService: unable to get trigger service for agent: SuperDomain|abc04g|WebSphere|corT2-abc04g/monitorServer2 [state=Connected, ipAddress=10.0.0.1, socketType=default, okToDisconnect=true, okToUnmount=true, okToAutoUnmount=true, supportsShutoff=true, shutoff=false, supportsTransactionTracing=true, supportedTransactionTracingFilterTypes=(0,1,2,3,4,5,6,7, dynamicInstrumentationFlags=0] 

com.wily.isengard.messageprimitives.TimeoutConnectionException: Service call to host {Unknown} timed out after 2000 ms com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.agent.beans.autotracing.IAutoTracingTriggerService.clearAllAutoTracingTriggers, v1, []} threadname PO:main Mailman 2 

at com.wily.isengard.messageprimitives.service.MessageServiceClient.blockOnResponse(MessageServiceClient.java:282) 

at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:163) 

at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:356) 

at com.sun.proxy.$Proxy163.clearAllAutoTracingTriggers(Unknown Source) 

at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl.agentAdded(AgentThresholdDeliveryServiceImpl.java:218) 

at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl$AutoTraceTriggerQuery.dataAdded(AgentThresholdDeliveryServiceImpl.java:632) 

at com.wily.isengard.ongoingquery.AbstractQueryServiceManager$NotifyAd 

 

Cause:

EM hangs due to large amount of Agents connecting at same time. Various temporary connection and performance issues may manifest this issue.

 

Environment:

EMs running APM 10.1 and 10.2 . This is fixed in APM 10.3

Resolution:

 This is fixed in APM 10.3. Earlier releases have hotfixes available --10.1 HF#25, 10.2 HF#24