Tomcat is not getting shutdown if the agent is attached - hangs
search cancel

Tomcat is not getting shutdown if the agent is attached - hangs


Article ID: 227932


Updated On:


DX Application Performance Management CA Application Performance Management (APM / Wily / Introscope) CA Application Performance Management Agent (APM / Wily / Introscope) DX APM SaaS


We have a problem regarding restart/shutdown of Tomcat application if the Agent is attached on Tomcat.

Normally, every day, in the mornings we are restarting all Tomcat servers. We found out that after starting Tomcat with the apm jar file, shutdown command or restart command doesn't end the process and is hanging on OS side. 

If we execute ps aux | grep Tomcat, we can see that "stop" parameter is visible in the end of the process, but the process does not terminate itself. We need to kill the process with the kill command afterwards.

Tomcat version is Apache Tomcat/7.0.109



Release : 21.6 (may affect other versions)

Component : Introscope


Problem identified as race condition between GC monitor and socket tracing

Thread dumps taken during the shutdown show following

Found one Java-level deadlock:
"Agent Execution":
  waiting to lock monitor 0x00007f9a70004ab8 (object 0x00000005e0cd3b88, a java.lang.Class),
  which is held by "main"
  waiting to lock monitor 0x00007f9ac0005b38 (object 0x00000005e0035378, a,
  which is held by "Agent Execution"


Two workarounds are available

1. Disable gcmonitor by setting


in the IntroscopeAgent profile.

2. Disable NIO socket tracings via the configured toggles pbd files by commenting out the following 4 toggle flags in file wily/releases/21.6/core/config/acc-master-toggles.pbd  e.g. add a # at the beginning of the entries as below

#TurnOn: NIOSocketTracing
#TurnOn: NIOSocketSummaryTracing
#TurnOn: NIOSelectorTracing
#TurnOn: NIODatagramSummaryTracing