Agent appears to cause Out of Memory (OOM) exception when instrumenting Websphere BPM

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

Description:

When monitoring WebSphere BPM, every time the project deploys their snapshot to the Process Server they see an OOM core dump to occur. Analysis of this dump shows thread making request to large amount of heap, 2GB+, which causes an OOM exception. Note as memory is never allocated the object does not show on heap dumps.

Solution:

APM engineering team reviewed and determined that the APM Agent itself is not allocating any memory, it is making a call to the stub (copied stack below) and that internally routes the call to the IBM IIOP protocol, which results in the large allocation request.

1XMCURTHDINFO  Current thread
NULL           ----------------------
3XMTHREADINFO      "WebContainer : 0" J9VMThread:0x00000000459CD700, j9thread_t:0x000001002CA522B0, java/lang/Thread:0x000000075E9EEE70, state:R,prio=5
3XMJAVALTHREAD      (java/lang/Thread getId:0x111, isDaemon:true)
3XMTHREADINFO1      (native thread ID:0x4C100FB, native priority:0x5,native policy:UNKNOWN)
3XMHEAPALLOC        Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3      Java callstack:
4XESTACKTRACE       at com/ibm/rmi/IOR.read(IOR.java:362(CompiledCode))
4XESTACKTRACE       at com/ibm/rmi/iiop/CDRReader.read_Object(CDRReader.java:1167)
4XESTACKTRACE       at com/ibm/rmi/iiop/CDRReader.read_abstract_interface(CDRReader.java:1602)
4XESTACKTRACE       at com/ibm/rmi/iiop/CDRReader.read_abstract_interface(CDRReader.java:1591)
4XESTACKTRACE       at com/lombardisoftware/server/ejb/tracking/_APIServicesHome_Stub.getEJBMetaData(_APIServicesHome_Stub.java)
4XESTACKTRACE       at com/wily/introscope/agent/servlethelper/EJBHelper.getEJBMetaData(EJBHelper.java:28)
4XESTACKTRACE       at com/wily/introscope/agent/trace/ejb/hc2/Ejb2StubNameFormatter.findInterfaceName(Ejb2StubNameFormatter.java:40)

The problem is known problem with the IBM JVM, see:-

http://www-304.ibm.com/support/docview.wss?uid=swg1PM46698

To overcome either upgrade the JVM or disable the EJB2StubTracing and EJB3StubTracing in toggles-full.pbd or toggles-typical.pbd. Simply add a # character to the TurnOn directive:-

 #TurnOn: EJB2StubTracing       
 #TurnOn: EJB3StubTracing 

Environment

Release:
Component: APMISP