A monitored JBoss application server frequently hangs with blocked threads with the following stack:
"default task-128" #314 prio=5 os_prio=0 tid=0x00007fb160194370 nid=0xe3b waiting for monitor entry [0x00007fb0c9360000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.wily.introscope.agent.trace.cas.SharedDataStructure5$SynchronizedCounter.increase(SharedDataStructure5.java:507)
- locked <0x0000000633f0f608> (a java.lang.Object)
at com.wily.introscope.agent.trace.cas.SharedDataStructure5$SynchronizedCounter.increase(SharedDataStructure5.java:583)
at com.wily.introscope.agent.trace.cas.SharedDataStructure5.increase(SharedDataStructure5.java:1108)
at com.wily.introscope.agent.trace.hc2.BlamePointTracer.increaseConcurrentInvocationCounter(BlamePointTracer.java:1345)
at com.wily.introscope.agent.trace.hc2.BlamePointTracer.doOnStartTrace(BlamePointTracer.java:1197)
at com.wily.introscope.agent.trace.cas.TransactionStructure.submitNextStartTransaction(TransactionStructure.java:431)
at com.wily.introscope.agent.trace.hc2.ASingleInstanceTracerFactoryHC.submitToTransactionStructureOnStartTrace(ASingleInstanceTracerFactoryHC.java:303)
at com.wily.introscope.agent.trace.hc2.BlamePointTracer.doStartBlamePointTrace(BlamePointTracer.java:434)
at com.wily.introscope.agent.trace.hc2.BlamePointTracer.ITracer_startTrace(BlamePointTracer.java:412)
at com.wily.introscope.agent.trace.InvocationData.IMethodTracer_startTrace(InvocationData.java:1374)
at com.wily.introscope.agent.trace.TracerAdministrator.loadTracer(TracerAdministrator.java:306)
at com.wily.introscope.agent.enterprise.EnterpriseAgent.IAgent_loadTracer(EnterpriseAgent.java:976)
at com.wily.introscope.agent.AgentShim.ProbeBuilderEntryPoint_loadTracer(AgentShim.java:1184)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java)
at com.liferay.portal.spring.transaction.DefaultTransactionExecutor._commit(DefaultTransactionExecutor.java:128)
at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.commit(DefaultTransactionExecutor.java:36)
........
Analysis of thread dump showed that one partiicular method was being accessed by multiple concurrent threads leading to a synchronisation issue as
numerous threads showed the following condition
"Connection.Sender [26.0.138.136:33771 - 26.0.138.105:7810],shared=liferay-channel-transport-0" #1881 prio=5 os_prio=0 tid=0x00007fb208053a10 nid=0x1bf0 waiting on condition [0x00007fb0be9ac000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000725a5e5b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
and others the following condition
"default task-183" #371 prio=5 os_prio=0 tid=0x00007fb228064090 nid=0xea4 waiting for monitor entry [0x00007fb0c5426000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.wily.introscope.agent.trace.cas.SharedDataStructure5$SynchronizedCounter.increase(SharedDataStructure5.java:507)
- waiting to lock <0x0000000633f0f608> (a java.lang.Object)
at com.wily.introscope.agent.trace.cas.SharedDataStructure5$SynchronizedCounter.increase(SharedDataStructure5.java:583)
at com.wily.introscope.agent.trace.cas.SharedDataStructure5.increase(SharedDataStructure5.java:1108)
at com.wily.introscope.agent.trace.hc2.BlamePointTracer.increaseConcurrentInvocationCounter(BlamePointTracer.java:1345)
at com.wily.introscope.agent.trace.hc2.BlamePointTracer.doOnStartTrace(BlamePointTracer.java:1197)
at com.wily.introscope.agent.trace.cas.TransactionStructure.submitNextStartTransaction(TransactionStructure.java:431)
at com.wily.introscope.agent.trace.hc2.ASingleInstanceTracerFactoryHC.submitToTransactionStructureOnStartTrace(ASingleInstanceTracerFactoryHC.java:303)
at com.wily.introscope.agent.trace.hc2.BlamePointTracer.doStartBlamePointTrace(BlamePointTracer.java:434)
at com.wily.introscope.agent.trace.hc2.BlamePointTracer.ITracer_startTrace(BlamePointTracer.java:412)
at com.wily.introscope.agent.trace.InvocationData.IMethodTracer_startTrace(InvocationData.java:1374)
at com.wily.introscope.agent.trace.TracerAdministrator.loadTracer(TracerAdministrator.java:306)
at com.wily.introscope.agent.enterprise.EnterpriseAgent.IAgent_loadTracer(EnterpriseAgent.java:976)
at com.wily.introscope.agent.AgentShim.ProbeBuilderEntryPoint_loadTracer(AgentShim.java:1184)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java)
at com.liferay.portal.spring.hibernate.PortletTransactionManager.commit(PortletTransactionManager.java:50)
....
Release : 10.7.0
Component : APM Agents
SkipClass: org.springframework.transaction.support.AbstractPlatformTransactionManager