Backend Call to an Oracle DB lead to JavaOutOfMemory with "wily" in the stack
IBM JDK 1.6
AgentNoRedefNoRetrans.jar and IntroscopeAgent.NoRedef.profile had been deployed correctly.
Release : 10.7.0
Component : APM Agents
On review of the heap dump, it was not necessarily an issue with agent objects in heap. The out of memory situation was caused by a lack of memory to create more threads
There were a lot of threads opened between the application and the database, nearly all of them were being instrumented by the agent.
Some example thread detail, note the presence of com/wily/introscope/agent/probe/net/ManagedSocketInputStream.read
"WebContainer : 252" J9VMThread:0x00007FA7C023EE00, j9thread_t:0x00007FA7C01DAC90, java/lang/Thread:0x00007FA82F2812E8, state:R, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x140B, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x52CB, native priority:0x5, native policy:UNKNOWN) 3XMTHREADINFO2 (native stack address range from:0x00007FA7FD3A2000, to:0x00007FA7FD3E3000, size:0x41000) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at java/net/SocketInputStream.socketRead0(Native Method) 4XESTACKTRACE at java/net/SocketInputStream.read(SocketInputStream.java:140(Compiled Code)) 4XESTACKTRACE at java/io/FilterInputStream.read(FilterInputStream.java:127(Compiled Code)) 4XESTACKTRACE at com/wily/introscope/agent/probe/net/ManagedSocketInputStream.read(ManagedSocketInputStream.java:216(Compiled Code)) 4XESTACKTRACE at oracle/net/ns/Packet.receive(Packet.java:308(Compiled Code)) 4XESTACKTRACE at oracle/net/ns/DataPacket.receive(DataPacket.java:106(Compiled Code)) 4XESTACKTRACE at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:324(Compiled Code)) 4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:268(Compiled Code)) 4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:190(Compiled Code)) 4XESTACKTRACE at oracle/net/ns/NetInputStream.read(NetInputStream.java:107(Compiled Code)) 4XESTACKTRACE at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124(Compiled Code)) 4XESTACKTRACE at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80(Compiled Code))
The presence of the instrumentation can slow transfer of data potentially increasing thread usage.
The suggestion was to disable all types of SocketTracing. In this instance, because the NoRedef profile was being used, ManagedSocketTracing and UDPTracing should be commented out in toggles-typical-NoRedef.pbd
#TurnOn: SocketTracing # NOTE: Only one of SocketTracing and ManagedSocketTracing should be 'on'. ManagedSocketTracing is provided to # enable pre 9.0 socket tracing. #TurnOn: ManagedSocketTracing #TurnOn: UDPTracing
No Socket communications or Backends|System metrics will be reported but there is a general recommendation to avoid any Socket Tracing in production monitoring.
To keep the socket tracing and investigate further, the associated directives are in java2.pbd. There are not many variables but you could test by enabling the TurnOn options commented out above and commenting the lines below one by one to see which one causes the issue - to then see if you do receive enough data from the options that are still enabled.