JBoss application hangs with threads blocked on APM agent class

Article Id: 196143

Status: Published

Updated On: 29-07-2020 03:49

Products:

CA Application Performance Management Agent (APM / Wily / Introscope) , CA Application Performance Management (APM / Wily / Introscope) , INTROSCOPE , DX Application Performance Management

Issue/Introduction:

A monitored JBoss application server frequently hangs with blocked threads with the following stack:




"default task-128" #314 prio=5 os_prio=0 tid=0x00007fb160194370 nid=0xe3b waiting for monitor entry [0x00007fb0c9360000]
   java.lang.Thread.State: BLOCKED (on object monitor)
 at com.wily.introscope.agent.trace.cas.SharedDataStructure5$SynchronizedCounter.increase(SharedDataStructure5.java:507)
 - locked <0x0000000633f0f608> (a java.lang.Object)
 at com.wily.introscope.agent.trace.cas.SharedDataStructure5$SynchronizedCounter.increase(SharedDataStructure5.java:583)
 at com.wily.introscope.agent.trace.cas.SharedDataStructure5.increase(SharedDataStructure5.java:1108)
 at com.wily.introscope.agent.trace.hc2.BlamePointTracer.increaseConcurrentInvocationCounter(BlamePointTracer.java:1345)
 at com.wily.introscope.agent.trace.hc2.BlamePointTracer.doOnStartTrace(BlamePointTracer.java:1197)
 at com.wily.introscope.agent.trace.cas.TransactionStructure.submitNextStartTransaction(TransactionStructure.java:431)
 at com.wily.introscope.agent.trace.hc2.ASingleInstanceTracerFactoryHC.submitToTransactionStructureOnStartTrace(ASingleInstanceTracerFactoryHC.java:303)
 at com.wily.introscope.agent.trace.hc2.BlamePointTracer.doStartBlamePointTrace(BlamePointTracer.java:434)
 at com.wily.introscope.agent.trace.hc2.BlamePointTracer.ITracer_startTrace(BlamePointTracer.java:412)
 at com.wily.introscope.agent.trace.InvocationData.IMethodTracer_startTrace(InvocationData.java:1374)
 at com.wily.introscope.agent.trace.TracerAdministrator.loadTracer(TracerAdministrator.java:306)
 at com.wily.introscope.agent.enterprise.EnterpriseAgent.IAgent_loadTracer(EnterpriseAgent.java:976)
 at com.wily.introscope.agent.AgentShim.ProbeBuilderEntryPoint_loadTracer(AgentShim.java:1184)
 at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java)
 at com.liferay.portal.spring.transaction.DefaultTransactionExecutor._commit(DefaultTransactionExecutor.java:128)
 at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.commit(DefaultTransactionExecutor.java:36)
........

Cause:

Analysis of thread dump showed that one partiicular method was being accessed by multiple concurrent threads leading to a synchronisation issue as

numerous threads showed the following condition

"Connection.Sender [26.0.138.136:33771 - 26.0.138.105:7810],shared=liferay-channel-transport-0" #1881 prio=5 os_prio=0 tid=0x00007fb208053a10 nid=0x1bf0 waiting on condition [0x00007fb0be9ac000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x0000000725a5e5b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

and others the following condition

"default task-183" #371 prio=5 os_prio=0 tid=0x00007fb228064090 nid=0xea4 waiting for monitor entry [0x00007fb0c5426000]
   java.lang.Thread.State: BLOCKED (on object monitor)
 at com.wily.introscope.agent.trace.cas.SharedDataStructure5$SynchronizedCounter.increase(SharedDataStructure5.java:507)
 - waiting to lock <0x0000000633f0f608> (a java.lang.Object)
 at com.wily.introscope.agent.trace.cas.SharedDataStructure5$SynchronizedCounter.increase(SharedDataStructure5.java:583)
 at com.wily.introscope.agent.trace.cas.SharedDataStructure5.increase(SharedDataStructure5.java:1108)
 at com.wily.introscope.agent.trace.hc2.BlamePointTracer.increaseConcurrentInvocationCounter(BlamePointTracer.java:1345)
 at com.wily.introscope.agent.trace.hc2.BlamePointTracer.doOnStartTrace(BlamePointTracer.java:1197)
 at com.wily.introscope.agent.trace.cas.TransactionStructure.submitNextStartTransaction(TransactionStructure.java:431)
 at com.wily.introscope.agent.trace.hc2.ASingleInstanceTracerFactoryHC.submitToTransactionStructureOnStartTrace(ASingleInstanceTracerFactoryHC.java:303)
 at com.wily.introscope.agent.trace.hc2.BlamePointTracer.doStartBlamePointTrace(BlamePointTracer.java:434)
 at com.wily.introscope.agent.trace.hc2.BlamePointTracer.ITracer_startTrace(BlamePointTracer.java:412)
 at com.wily.introscope.agent.trace.InvocationData.IMethodTracer_startTrace(InvocationData.java:1374)
 at com.wily.introscope.agent.trace.TracerAdministrator.loadTracer(TracerAdministrator.java:306)
 at com.wily.introscope.agent.enterprise.EnterpriseAgent.IAgent_loadTracer(EnterpriseAgent.java:976)
 at com.wily.introscope.agent.AgentShim.ProbeBuilderEntryPoint_loadTracer(AgentShim.java:1184)
 at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java)
 at com.liferay.portal.spring.hibernate.PortletTransactionManager.commit(PortletTransactionManager.java:50)

....

Environment:

Release : 10.7.0

Component : APM Agents

Resolution:

Solution in this case was to skip the monitored class by creating a file called skips.pbd in the agent hotdeploy folder with the following content:



SkipClass: org.springframework.transaction.support.AbstractPlatformTransactionManager