APM Introscope Java Agent causing High CPU Usage
search cancel

APM Introscope Java Agent causing High CPU Usage

book

Article ID: 116710

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) INTROSCOPE

Issue/Introduction

After enabling CA APM Introscope Agent 10.2.0.27 to monitor our IBM WebSphere Portal 8.5 environment (with WAS 8.5.5.11 and IBM Java 7.1.3.60 / Suse Linux Enterprise Server 12), we experiencing very high CPU usage in production due to the Introscope agent, thread dumps indicates that the high CPU symptom leading to the heartbeat of the agent is caused by the Java Agent.

Environment

Any Java agent version

Cause

APM Java agent uses PMIRegistry API to read PMI module data. The dump points that PMIRegistry.get() method taking more CPU. 
The issue is not related to a bug in the  agent code, Agent uses PMIRegistry API and Webshpere API takes more CPU especially java/util/concurrent/ConcurrentHashMap$Segment.get method of JDK. 

Below thread to illustrate the situation:

"Agent Heartbeat" J9VMThread:0x0000000002146600, j9thread_t:0x00007F8CA0D13EA0, java/lang/Thread:0x0000000100CEC318, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0xF, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x2D1A, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x00007F8C98D42000, to:0x00007F8C98D83000, size:0x41000)
3XMCPUTIME               CPU usage total: 19053.985865272 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=822757336 (0x310A47D8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/util/concurrent/ConcurrentHashMap$Segment.get(ConcurrentHashMap.java:443(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ConcurrentHashMap.get(ConcurrentHashMap.java:891(Compiled Code))
4XESTACKTRACE                at java/io/ObjectStreamClass.lookup(ObjectStreamClass.java:356(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1586(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1547(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1586(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1547(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:368(Compiled Code))
4XESTACKTRACE                at java/util/ArrayList.writeObject(ArrayList.java:754(Compiled Code))
4XESTACKTRACE                at sun/reflect/GeneratedMethodAccessor17.invoke(Bytecode PC:40(Compiled Code))
4XESTACKTRACE                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56(Compiled Code))
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code))
4XESTACKTRACE                at java/io/ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1110(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1534(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1586(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1547(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:368(Compiled Code))
4XESTACKTRACE                at java/util/HashMap.writeObject(HashMap.java:1141(Compiled Code))
4XESTACKTRACE                at sun/reflect/GeneratedMethodAccessor18.invoke(Bytecode PC:40(Compiled Code))
4XESTACKTRACE                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56(Compiled Code))
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code))
4XESTACKTRACE                at java/io/ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1110(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1534(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:368(Compiled Code))
4XESTACKTRACE                at java/util/Hashtable.writeObject(Hashtable.java:1042(Compiled Code))
4XESTACKTRACE                at sun/reflect/GeneratedMethodAccessor15.invoke(Bytecode PC:40(Compiled Code))
4XESTACKTRACE                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56(Compiled Code))
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code))
4XESTACKTRACE                at java/io/ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1110(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1534(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:368(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/session/WsSessionContextMBean.getSessionsInfo(WsSessionContextMBean.java:103(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/session/PMISessionStatistics$1.getRuntimeInfo(PMISessionStatistics.java:68(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/modules/ServletSessionsModule$SessionObjSize.updateStatistic(ServletSessionsModule.java:486(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/data/SpdStatExternal.updateExternal(SpdStatExternal.java:77(Compiled Code))
4XESTACKTRACE                at com/ibm/websphere/pmi/server/PmiAbstractModule.update(PmiAbstractModule.java:1395(Compiled Code))
4XESTACKTRACE                at com/ibm/websphere/pmi/server/PmiAbstractModule.updateStatistics(PmiAbstractModule.java:1418(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/ModuleItem.updateStatisticsForStatsTree(ModuleItem.java:588(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/ModuleItem.updateStatisticsForStatsTree(ModuleItem.java:599(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/ModuleItem.getStats(ModuleItem.java:717(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/PmiRegistry.getStats(PmiRegistry.java:930(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/PmiRegistry.get(PmiRegistry.java:885(Compiled Code))
4XESTACKTRACE                at com/wily/introscope/agent/pmi/PMIDataPoller$PMIModule.pollPMIForUpdates(PMIDataPoller.java:334(Compiled Code))
4XESTACKTRACE                at com/wily/introscope/agent/pmi/PMIDataPoller.ITimestampedRunnable_execute(PMIDataPoller.java:201(Compiled Code))
..

Resolution

On the Java Agent side:

a) Disable PMI feature: introscope.agent.pmi.enable=fase. 
You need to restart the JVM/Appserver

b) Keep PMI feature enabled but reduce PMI collection
Disable all Websphere PMI Modules in the IntroscopeAgent.profile and enable them 1 by 1 in order to identify the best settings, for example:
..
introscope.agent.pmi.enable.threadPoolModule=false 
introscope.agent.pmi.enable.servletSessionsModule=false 
introscope.agent.pmi.enable.connectionPoolModule=false 
introscope.agent.pmi.enable.j2cModule=true 
..
See: https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/implementing-agents/java-agent/java-agent-properties.html#concept.dita_e3b0b5e4c888550feb742b0f947b9fa2e2fd716d_WebSpherePMIProperties


On the Appserver side:

a) Upgrade the Appserver JAVA version

b) The WAS Admin Console provides more granular control over which PMI metrics are reported for all consumers including Introscope 
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/implementing-agents/java-agent/install-the-java-agent/configure-application-server-to-use-the-java-agent/websphere-application-server.html#concept.dita_645c902f8a646ca22bbb2a8a15fdf89b4fc44b26_EnabletheCollectionofIBMWebSpherePMIMetrics
​

Additional Information

All APM compatibility guides are available from:
https://techdocs.broadcom.com/us/product-content//status/compatibility-matrix/application-performance-management-compatibility-guide.html?r=2
Powered by Wolken Software