Processes are getting struck due to BEANUTILS defect

Products

• Clarity PPM SaaS
• Clarity PPM On Premise

Issue/Introduction

The processes (using gel scripts) are getting stuck especially when the load on the process engine is high.

Cause

This is caused by an existing defect in apache commons-beanutils library

Here is the link to the defect - https://issues.apache.org/jira/browse/BEANUTILS-509

To confirm if the processes are stuck due to this defect, generate a thread dump while the issue is ongoing and threads of the "Custom script execution pool" (Thread names start with this pool name) are stuck and each one of the stacktraces will contain the following line - org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47)

 

"Custom script execution pool-889-thread-13" #98977 prio=5 os_prio=0 cpu=12347.46ms elapsed=1049037.13s tid=0x00007fc188007000 nid=0x4b35b waiting for monitor entry  [0x00007fc08b7f5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
 at java.util.WeakHashMap.expungeStaleEntries([email protected]/WeakHashMap.java:319)
 - waiting to lock <0x000000068efa1da8> (a java.lang.ref.ReferenceQueue)
 at java.util.WeakHashMap.getTable([email protected]/WeakHashMap.java:350)
 at java.util.WeakHashMap.get([email protected]/WeakHashMap.java:398)
 at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426)
 at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405)
 at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230)
 at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79)
 at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61)
 at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47)
 at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:225)
 at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:96)
 at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:187)
 at org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:43)
 at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:248)
 at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:96)
 at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:187)
 at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:42)
 at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:248)
 at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:96)
 at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:187)
 at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:39)
 at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:248)
 at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:96)
 at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:187)
 at com.niku.union.gel.tags.ScriptTag.doTag(ScriptTag.java:20)
 at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:248)
 at com.niku.union.gel.GELScript.run(GELScript.java:58)
 at com.niku.union.gel.GELController.invoke(GELController.java:79)
 at com.niku.bpm.services.ExecuteCustomAction.run(ExecuteCustomAction.java:217)
 at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
 at java.lang.Thread.run([email protected]/Thread.java:829)

   Locked ownable synchronizers:
 - <0x0000000697062810> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Resolution

This is a known apache defect - https://issues.apache.org/jira/browse/BEANUTILS-509

The workaround is to restart the BG services

The fix for this will be included in 16.1.3

Additional Information

Here is the link to a defect logged in the Jenkins project where the above beanutils defect is mentioned as the root cause - https://issues.jenkins.io/browse/JENKINS-60997?attachmentOrder=desc