Processes stuck / hung - troubleshooting

Products

• Clarity PPM On Premise
• Clarity PPM SaaS

Issue/Introduction

This document provides some tips on troubleshooting process stuck/hung issues as well as links to KBs on known causes.

Resolution

Table of Contents

• Tips for optimizing performance health of processes
• Log troubleshooting
• Symptom Troubleshooting
  • Not Starting
  • Stuck with Error
  • Gel Processes stuck
  • Stuck at Evaluate Pre or Post-Condition
  • Stuck at the Review step
  • Aborting / Aborted Status Issues
  • Fails with error
  • System Actions
  • Process Engine Down / Stuck
  • Completion Message not displaying
  • Notifications not sent
• If Broadcom support assistance is needed

Tips for optimizing performance health of processes

1. Delete old processes in the Done or Aborted state.
2. Retry or Abort and Delete old processes in the Error state. See more details at Process engine - keeping it healthy
3. If the issue is related to a single process that performs some type of data load, it might be slow because of multiple instances of the process running at the same time. In this scenario checking the "Do not start a new process if one is already running" flag on the process definition will help.

Log troubleshooting

Review of the bg-ca.logs / bg-system.logs can assist in determining a root cause / fix. Additionally, enabling process debugging (see KB: Process debugging - Enabling for Troubleshooting) may help further in troubleshooting.

If an error is found in the logs, searching our knowledge base on that error may help in determining the root cause / fix needed.

Symptom Troubleshooting

Not Starting

• Processes do not start intermittently - The below error message is seen in the bg-ca.logs:

  Error message: [CA Clarity][SQLServer JDBC Driver][SQLServer]Transaction (Process ID 65) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
  
  

Stuck with Error

• Database connection pool on Custom Processes
• Process stuck with ODF-0022: Invalid expression

Gel Processes stuck

• Procesess that use GEL are getting stuck

Stuck at Evaluate Pre or Post-Condition

• Processes Stuck in Ready to Evaluate Pre or Post-Condition

Stuck at the Review step

• Idea Approval Process Stuck at Review Step

Aborting / Aborted Status Issues

• Unable to Delete Process Instances in the 'Aborted' Status 
• Processes stuck in Aborting status for long time - How to set them to aborted

Fails with error

• Process BPM-0543 error "'Subject' is too large. Maximum size is 128"
• Processes erroring with BPM-0526 error in Clarity PPM due to tablespace issue
• Process with XOG fails with value too large error
• Error: Unable to initiate the process XXX, please contact the administrator.
• Process With GEL Fails With Error - Not Authorized To Create An Object Instance
• Process error: "BPM-0545: An error occurred when executing custom action" occurring intermittently

System Actions

• System Actions in Processes erroring based on custom investments after upgrade to 15.9.2

Process Engine Down / Stuck

Check for processes in 'Aborting' status (Administration > Data Administration > Processes). If processes are stuck in aborting status, see Aborting / Aborted Status issues section above. 

• Clarity Process Engine down / error in BG logs
• Processes failing / Process Engines in Stopped Status - error in bg-ca.log references "Cannot get DB Connection"
• Process Engine shuts down repeatedly - Log Error message: "[CA Clarity] [Oracle JDBC Driver] [Oracle] ORA-01555: too old snapshot: rollback segment number 29..."
• Process engine not working with bg error: "heart beat is old"

Completion Message not displaying

• Processes Not Showing Completion

Notifications not sent

• GEL Processes are not sending email notifications

If Broadcom support assistance is needed

The below details can aid support in troubleshooting stuck process issues:

1. Is it just certain processes that are stuck?
2. How long has the issue been occurring?
3. Has there been any recent upgrade, a new patch applied or known change in the environment?
4. Do out of the box processes run okay? 
   Tip: You can check Administration-Processes->Initiated, and filter on Status of Error in case the processes are erroring versus stuck.
   • If you click on the icon on the Messages column, it will give more details of the error, and sometimes give you the ability to correct the issue from there. (Example, if there's no one to assign an action item to in the process, it gives a reassign link).
   • Some objects give specific details as to where the issue is located, for example Timesheets.
5. Do jobs appear to be running okay? 
6. If On Premise, provide a copy of the bg-ca.logs and bg-system.logs for the time frame the issue occurred. Enabling process debugging (see KB: Process debugging - Enabling for Troubleshooting) may help in further investigation
7. Screenshots of the 2 below pages: 

1. • Administration->Data Administration->Process Engine page. On Premise customers can also run the following query and send the results: 

select * from BPM_RUN_PROCESS_ENGINES order by heart_beat desc

• • From the Process Engine Page, click on the link to the process engine and take a screenshot of that page as well

Additional Information

See Process engine - keeping it healthy

See Monitoring process engine activity in Clarity