Process Instances Stuck In Start Never Finishing

Products

• Clarity PPM SaaS
• Clarity PPM On Premise

Issue/Introduction

Process instances with Start step of calling a GEL script are not finishing. They are stuck in the this start step. 
They do not include any sleep function and should complete within a second.

Restarting the BG service does not work either.

Environment

All Versions

Resolution

1. Check if the BG system logs has any error related to the process engine, such as "Exception in thread "Custom script execution pool-9-thread-7" java.lang.NullPointerException"
If there is no error, proceed to next step.

2. Have the server team produce 3 thread dumps, taken 5 minutes apart on the server running the BG service.
This is for reference and to find out if there is any deadlock.

Review the thread dumps for deadlocks.

If there is no deadlock, create a process that calls the Clean User Sessions job.
Run the job(Execute A Process) with the job as a parameter.

3. Check if there is a specific issue with a custom process.
Is there a specific query causing a performance issue?
Is there any maintenance that can be performed on the GEL script or tables?

4. Restart the BG service