Clarity Processes getting stuck / slow to transition due to multiple slow GELs

Products

• Clarity PPM SaaS
• Clarity PPM On Premise

Issue/Introduction

• Processes are stuck, there is process contention, the processes that have just 1 GEL script which takes 100 ms will stay stuck in Running and not move to Complete until 8-10 min later
• Sometimes the entire system gets overloaded with processes because of this and we have to restart BG (sometimes few times) to get the processes to move along again
• We have many different processes with GEL scripts, on multiple objects and tables
• Some of the GEL scripts are large and run over 10 minutes

Environment

Release : Any

Resolution

• In the application we have a maximum of 15 threads that can run Custom Action Execution Scripts (GEL), this is a setting that is separate from the pipelines and is not exposed in database or the configuration.
• This means that if those 15 threads in BG are running the Custom Script Execution for 10 minutes each for each GEL, no other process can run and they will wait appearing stuck.
• The threads will just run the action until they are completed, then pick up the next actions.
• If by chance another one of the slow processes comes up, the bottleneck may continue.
• Restarting BG may change the process order and allow the other processes to be picked up first.
• To resolve the problem: We recommend redesigning the slow GEL scripts that take >10 min or making sure there is just few running at once, never over 5 at a time, ideally 1 slow running process at a given time