Process bottlenecks on too many process instances started at once

Products

• Clarity PPM SaaS
• Clarity PPM On Premise

Issue/Introduction

You have implemented a new workflow and started multiple thousand process instances at once. They are not progressing and restarting BG does not help, are they stuck? 

Environment

Any Supported Clarity Releases

Resolution

• Broadcom does not recommend to start many thousand processes, specifically GEL processes at once
• All custom workflows must be thoroughly tested on a lower environment, including for concurrency

Best Practices: 

• Batching: Process the updates in smaller batches (100-300 instances at a time) to prevent hitting the database at once
• REST API: Transition from direct database update on database tables to using the REST API for updates. This is considered a safer, more throttled update
• Workflow Redesign: Use a single master process loop through all instances for the update, rather than spinning off individual subprocesses for each. This would reduce the number of concurrent process instances and the risk of running into bottlenecks

For GELs 

• If your GEL script takes longer, i.e. over 1 min, you also risk into additional bottlenecks
• Slow GELs: Clarity Processes getting stuck / slow to transition due to multiple slow GELs
• Also you may hit Connection Timeouts