We have a problem regarding restart/shutdown of Tomcat application if the Agent is attached on Tomcat.
Normally, every day, in the mornings we are restarting all Tomcat servers. We found out that after starting Tomcat with the apm jar file, shutdown command or restart command doesn't end the process and is hanging on OS side.
If we execute ps aux | grep Tomcat, we can see that "stop" parameter is visible in the end of the process, but the process does not terminate itself. We need to kill the process with the kill command afterwards.
Tomcat version is Apache Tomcat/7.0.109
Release : 21.6 (may affect other versions)
Component : Introscope
Problem identified as race condition between GC monitor and socket tracing
Thread dumps taken during the shutdown show following
Found one Java-level deadlock:
=============================
"Agent Execution":
waiting to lock monitor 0x00007f9a70004ab8 (object 0x00000005e0cd3b88, a java.lang.Class),
which is held by "main"
"main":
waiting to lock monitor 0x00007f9ac0005b38 (object 0x00000005e0035378, a com.wily.util.properties.hot.ConfigurationManager),
which is held by "Agent Execution"
Two workarounds are available
1. Disable gcmonitor by setting
introscope.agent.gcmonitor.enable=false
in the IntroscopeAgent profile.
2. Disable NIO socket tracings via the configured toggles pbd files by commenting out the following 4 toggle flags in file wily/releases/21.6/core/config/acc-master-toggles.pbd e.g. add a # at the beginning of the entries as below
#TurnOn: NIOSocketTracing
#TurnOn: NIOSocketSummaryTracing
#TurnOn: NIOSelectorTracing
#TurnOn: NIODatagramSummaryTracing