Introscope agent app crash socket backend
search cancel

Introscope agent app crash socket backend


Article ID: 191143


Updated On:


CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management


Backend Call to an Oracle DB lead to JavaOutOfMemory with "wily" in the stack


WebSphere 7.0.33

AgentNoRedefNoRetrans.jar and IntroscopeAgent.NoRedef.profile had been deployed correctly.


Release : 10.7.0

Component : APM Agents


On review of the heap dump, it was not necessarily an issue with agent objects in heap.
The out of memory situation was caused by a lack of memory to create more threads

There were a lot of threads opened between the application and the database, nearly all of them were being instrumented by the agent.

Some example thread detail, note the presence of com/wily/introscope/agent/probe/net/

"WebContainer : 252" J9VMThread:0x00007FA7C023EE00, j9thread_t:0x00007FA7C01DAC90, java/lang/Thread:0x00007FA82F2812E8, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x140B, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x52CB, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range from:0x00007FA7FD3A2000, to:0x00007FA7FD3E3000, size:0x41000)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/net/SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE                at java/net/ Code))
4XESTACKTRACE                at java/io/ Code))
4XESTACKTRACE                at com/wily/introscope/agent/probe/net/ Code))
4XESTACKTRACE                at oracle/net/ns/Packet.receive( Code))
4XESTACKTRACE                at oracle/net/ns/DataPacket.receive( Code))
4XESTACKTRACE                at oracle/net/ns/NetInputStream.getNextPacket( Code))
4XESTACKTRACE                at oracle/net/ns/ Code))
4XESTACKTRACE                at oracle/net/ns/ Code))
4XESTACKTRACE                at oracle/net/ns/ Code))
4XESTACKTRACE                at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket( Code))
4XESTACKTRACE                at oracle/jdbc/driver/ Code))

The presence of the instrumentation can slow transfer of data potentially increasing thread usage.


The suggestion was to disable all types of SocketTracing.
In this instance, because the NoRedef profile was being used, ManagedSocketTracing and UDPTracing should be commented out in toggles-typical-NoRedef.pbd

# Network Configuration
# ================

#TurnOn: SocketTracing
# NOTE: Only one of SocketTracing and ManagedSocketTracing should be 'on'.  ManagedSocketTracing is provided to
# enable pre 9.0 socket tracing. 
#TurnOn: ManagedSocketTracing
#TurnOn: UDPTracing

No Socket communications or Backends|System metrics will be reported but there is a general recommendation to avoid any Socket Tracing in production monitoring.

Additional Information

To keep the socket tracing and investigate further, the associated directives are in java2.pbd.
There are not many variables but you could test by enabling the TurnOn options commented out above and commenting the lines below one by one to see which one causes the issue - to then see if you do receive enough data from the options that are still enabled.

SubstituteClassIfFlagged: ManagedSocketTracing
SubstituteClassIfFlagged: ManagedSocketTracing
SubstituteClassIfFlagged: UDPTracing

In this situation as well, this IBM article about tuning connection pooling may also be useful