Process Automation instances in Waiting and Blocked states

Products

• CA Process Automation Base

Issue/Introduction

In a Service Management environment with Service Desk and Process Automation (cluster) integrated, customers are noticing abnormal behavior on Process Automation side when changes are performed on the Service Desk side. For example, if for some reason Service Desk servers have to be restarted, instances start being at Waiting and Blocked states on Process Automation side. On the other hand, new instances work fine and only a few older instances stay stuck on those states.

Message in c2o.log:

ERROR [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [49-076f208c44fe] Error While Handling a response. Response is
org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.optinuity.c2o.bean.C2ORunTimeInstanceArchive#412963]

Process Automation interface:

Environment

Process Automation 4.3.X

Cause

There are many reasons for the message "Error While Handling a response", but according to the symptom customers are facing, it seems Process Automation was not properly stopped/restarted during a change request on Service Desk side.

Resolution

If Process Automation is configured as cluster, it is important to stop both Process Automation servers before any intervention on Service Desk side. When finished, Process Automation servers must be started with an interval between them. It is well known Process Automation takes some time to be up and ready and because of that, it is strongly recommended to make sure the Node 1 is fully available before starting the Node 2.

Steps to be taken:

1. Shut down Process Automation nodes;
2. Make sure there is no remaining opened connection in the database - if you find some, kill all of them;
3. Set INFO level logging in case more detail for troubleshooting;
4. Start Node 1 and monitor to see if instances begin to move, if the instances are moving go ahead and start Node 2. If no instances are moving let it sit for 10 minutes, then gather the logs for review. 

IMPORTANT: Do not forget to set log level to ERROR after resolving the issue. 

Additional Information

CA Process Automation: Increase logging levels, file size and rollover for PAM logs 
https://comm.support.ca.com/kb/ca-process-automation-increase-logging-levels-file-size-and-rollover-for-pam-logs/kb000009347