Tomcat is not getting shutdown if the agent is attached - hangs

Products

• DX Application Performance Management
• CA Application Performance Management (APM / Wily / Introscope)
• CA Application Performance Management Agent (APM / Wily / Introscope)
• DX APM SaaS

Issue/Introduction

We have a problem regarding restart/shutdown of Tomcat application if the Agent is attached on Tomcat.

Normally, every day, in the mornings we are restarting all Tomcat servers. We found out that after starting Tomcat with the apm jar file, shutdown command or restart command doesn't end the process and is hanging on OS side. 

If we execute ps aux | grep Tomcat, we can see that "stop" parameter is visible in the end of the process, but the process does not terminate itself. We need to kill the process with the kill command afterwards.

Tomcat version is Apache Tomcat/7.0.109

 

Environment

Release : 21.6 (may affect other versions)

Component : Introscope

Cause

Problem identified as race condition between GC monitor and socket tracing

Thread dumps taken during the shutdown show following

Found one Java-level deadlock:
=============================
"Agent Execution":
  waiting to lock monitor 0x00007f9a70004ab8 (object 0x00000005e0cd3b88, a java.lang.Class),
  which is held by "main"
"main":
  waiting to lock monitor 0x00007f9ac0005b38 (object 0x00000005e0035378, a com.wily.util.properties.hot.ConfigurationManager),
  which is held by "Agent Execution"

Resolution

Two workarounds are available

1. Disable gcmonitor by setting

introscope.agent.gcmonitor.enable=false

in the IntroscopeAgent profile.

2. Disable NIO socket tracings via the configured toggles pbd files by commenting out the following 4 toggle flags in file wily/releases/21.6/core/config/acc-master-toggles.pbd  e.g. add a # at the beginning of the entries as below

#TurnOn: NIOSocketTracing
#TurnOn: NIOSocketSummaryTracing
#TurnOn: NIOSelectorTracing
#TurnOn: NIODatagramSummaryTracing