The processes (using gel scripts) are getting stuck especially when the load on the process engine is high.
This is caused by an existing defect in apache commons-beanutils library
Here is the link to the defect - https://issues.apache.org/jira/browse/BEANUTILS-509
To confirm if the processes are stuck due to this defect, generate a thread dump while the issue is ongoing and threads of the "Custom script execution pool" (Thread names start with this pool name) are stuck and each one of the stacktraces will contain the following line - org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47)
"Custom script execution pool-889-thread-13" #98977 prio=5 os_prio=0 cpu=12347.46ms elapsed=1049037.13s tid=0x00007fc188007000 nid=0x4b35b waiting for monitor entry [0x00007fc08b7f5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.WeakHashMap.expungeStaleEntries([email protected]/WeakHashMap.java:319)
- waiting to lock <0x000000068efa1da8> (a java.lang.ref.ReferenceQueue)
at java.util.WeakHashMap.getTable([email protected]/WeakHashMap.java:350)
at java.util.WeakHashMap.get([email protected]/WeakHashMap.java:398)
at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426)
at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405)
at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230)
at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79)
at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61)
at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:225)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:96)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:187)
at org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:43)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:248)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:96)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:187)
at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:42)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:248)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:96)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:187)
at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:39)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:248)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:96)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:187)
at com.niku.union.gel.tags.ScriptTag.doTag(ScriptTag.java:20)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:248)
at com.niku.union.gel.GELScript.run(GELScript.java:58)
at com.niku.union.gel.GELController.invoke(GELController.java:79)
at com.niku.bpm.services.ExecuteCustomAction.run(ExecuteCustomAction.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:829)
Locked ownable synchronizers:
- <0x0000000697062810> (a java.util.concurrent.ThreadPoolExecutor$Worker)
This is a known apache defect - https://issues.apache.org/jira/browse/BEANUTILS-509
The workaround is to restart the BG services
The fix for this will be included in 16.1.3
Here is the link to a defect logged in the Jenkins project where the above beanutils defect is mentioned as the root cause - https://issues.jenkins.io/