Introscope agent app crash socket backend
Article ID: 191143
Updated On:
Products
CA Application Performance Management Agent (APM / Wily / Introscope)
CA Application Performance Management (APM / Wily / Introscope)
INTROSCOPE
DX Application Performance Management
Issue/Introduction
Backend Call to an Oracle DB lead to JavaOutOfMemory with "wily" in the stack
IBM JDK 1.6
WebSphere 7.0.33
AgentNoRedefNoRetrans.jar and IntroscopeAgent.NoRedef.profile had been deployed correctly.
IBM JDK 1.6
WebSphere 7.0.33
AgentNoRedefNoRetrans.jar and IntroscopeAgent.NoRedef.profile had been deployed correctly.
Environment
Release : 10.7.0
Component : APM Agents
Cause
On review of the heap dump, it was not necessarily an issue with agent objects in heap.
The out of memory situation was caused by a lack of memory to create more threads
There were a lot of threads opened between the application and the database, nearly all of them were being instrumented by the agent.
Some example thread detail, note the presence of com/wily/introscope/agent/probe/net/ManagedSocketInputStream.read
"WebContainer : 252" J9VMThread:0x00007FA7C023EE00, j9thread_t:0x00007FA7C01DAC90, java/lang/Thread:0x00007FA82F2812E8, state:R, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x140B, isDaemon:true)
3XMTHREADINFO1 (native thread ID:0x52CB, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2 (native stack address range from:0x00007FA7FD3A2000, to:0x00007FA7FD3E3000, size:0x41000)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/net/SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE at java/net/SocketInputStream.read(SocketInputStream.java:140(Compiled Code))
4XESTACKTRACE at java/io/FilterInputStream.read(FilterInputStream.java:127(Compiled Code))
4XESTACKTRACE at com/wily/introscope/agent/probe/net/ManagedSocketInputStream.read(ManagedSocketInputStream.java:216(Compiled Code))
4XESTACKTRACE at oracle/net/ns/Packet.receive(Packet.java:308(Compiled Code))
4XESTACKTRACE at oracle/net/ns/DataPacket.receive(DataPacket.java:106(Compiled Code))
4XESTACKTRACE at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:324(Compiled Code))
4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:268(Compiled Code))
4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:190(Compiled Code))
4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:107(Compiled Code))
4XESTACKTRACE at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124(Compiled Code))
4XESTACKTRACE at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80(Compiled Code))
The presence of the instrumentation can slow transfer of data potentially increasing thread usage.
The out of memory situation was caused by a lack of memory to create more threads
There were a lot of threads opened between the application and the database, nearly all of them were being instrumented by the agent.
Some example thread detail, note the presence of com/wily/introscope/agent/probe/net/ManagedSocketInputStream.read
"WebContainer : 252" J9VMThread:0x00007FA7C023EE00, j9thread_t:0x00007FA7C01DAC90, java/lang/Thread:0x00007FA82F2812E8, state:R, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x140B, isDaemon:true)
3XMTHREADINFO1 (native thread ID:0x52CB, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2 (native stack address range from:0x00007FA7FD3A2000, to:0x00007FA7FD3E3000, size:0x41000)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/net/SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE at java/net/SocketInputStream.read(SocketInputStream.java:140(Compiled Code))
4XESTACKTRACE at java/io/FilterInputStream.read(FilterInputStream.java:127(Compiled Code))
4XESTACKTRACE at com/wily/introscope/agent/probe/net/ManagedSocketInputStream.read(ManagedSocketInputStream.java:216(Compiled Code))
4XESTACKTRACE at oracle/net/ns/Packet.receive(Packet.java:308(Compiled Code))
4XESTACKTRACE at oracle/net/ns/DataPacket.receive(DataPacket.java:106(Compiled Code))
4XESTACKTRACE at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:324(Compiled Code))
4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:268(Compiled Code))
4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:190(Compiled Code))
4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:107(Compiled Code))
4XESTACKTRACE at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124(Compiled Code))
4XESTACKTRACE at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80(Compiled Code))
The presence of the instrumentation can slow transfer of data potentially increasing thread usage.
Resolution
The suggestion was to disable all types of SocketTracing.
In this instance, because the NoRedef profile was being used, ManagedSocketTracing and UDPTracing should be commented out in toggles-typical-NoRedef.pbd
#######################
# Network Configuration
# ================
#TurnOn: SocketTracing
# NOTE: Only one of SocketTracing and ManagedSocketTracing should be 'on'. ManagedSocketTracing is provided to
# enable pre 9.0 socket tracing.
#TurnOn: ManagedSocketTracing
#TurnOn: UDPTracing
No Socket communications or Backends|System metrics will be reported but there is a general recommendation to avoid any Socket Tracing in production monitoring.
In this instance, because the NoRedef profile was being used, ManagedSocketTracing and UDPTracing should be commented out in toggles-typical-NoRedef.pbd
#######################
# Network Configuration
# ================
#TurnOn: SocketTracing
# NOTE: Only one of SocketTracing and ManagedSocketTracing should be 'on'. ManagedSocketTracing is provided to
# enable pre 9.0 socket tracing.
#TurnOn: ManagedSocketTracing
#TurnOn: UDPTracing
No Socket communications or Backends|System metrics will be reported but there is a general recommendation to avoid any Socket Tracing in production monitoring.
Additional Information
To keep the socket tracing and investigate further, the associated directives are in java2.pbd.
There are not many variables but you could test by enabling the TurnOn options commented out above and commenting the lines below one by one to see which one causes the issue - to then see if you do receive enough data from the options that are still enabled.
SubstituteClassIfFlagged: ManagedSocketTracing java.net.Socket com.wily.introscope.agent.probe.net.ManagedSocket
SubstituteClassIfFlagged: ManagedSocketTracing java.net.ServerSocket com.wily.introscope.agent.probe.net.ManagedServerSocket
SubstituteClassIfFlagged: UDPTracing java.net.DatagramSocket com.wily.introscope.agent.probe.net.ManagedDatagramSocket
In this situation as well, this IBM article about tuning connection pooling may also be useful
https://www.ibm.com/support/pages/troubleshooting-connection-pooling-j2c-problems-websphere-application-server
There are not many variables but you could test by enabling the TurnOn options commented out above and commenting the lines below one by one to see which one causes the issue - to then see if you do receive enough data from the options that are still enabled.
SubstituteClassIfFlagged: ManagedSocketTracing java.net.Socket com.wily.introscope.agent.probe.net.ManagedSocket
SubstituteClassIfFlagged: ManagedSocketTracing java.net.ServerSocket com.wily.introscope.agent.probe.net.ManagedServerSocket
SubstituteClassIfFlagged: UDPTracing java.net.DatagramSocket com.wily.introscope.agent.probe.net.ManagedDatagramSocket
In this situation as well, this IBM article about tuning connection pooling may also be useful
https://www.ibm.com/support/pages/troubleshooting-connection-pooling-j2c-problems-websphere-application-server
Feedback
Yes
No
Powered by
