Deadlocks during workflow-execution in combination with different limited queues.

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

Error Message :
U0003594 UCUDB Ret: '6' opcode: 'EXEC' SQL Stmnt: 'UPDATE FIFO SET FIFO_Nr = FIFO_Nr + 1 WHERE FIFO_Name = ?'

If more than one task is started in parallel within a workflow and every task is started in a different LIMITED queue, this might lead to Deadlocks and you will see errors similar to these:
 
41 - 20151023/064336.555 - U0000006 DEADLOCK 
41 - 20151023/064336.556 - U0003594 UCUDB Ret: '6' opcode: 'EXEC' SQL Stmnt: 'UPDATE FIFO SET FIFO_Nr = FIFO_Nr + 1 WHERE FIFO_Name = ?' 
41 - 20151023/064336.556 - U0003524 UCUDB: ===> Time critical DB call! OPC: 'EXEC' time: '26:418.780.999' 
41 - 20151023/064336.556 - U0029131 UCUDB - List of bind parameters: 
41 - 20151023/064336.556 - U0029132 UCUDB: 'FIFO_Name = QUEUE.TD.TD2.WRAPPER' 
41 - 20151023/064336.556 - U0003525 UCUDB: ===> 'UPDATE FIFO SET FIFO_Nr = FIFO_Nr + 1 WHERE FIFO_Name = ?' 

Basically, what occurs is:

1.  A task ends and has 2+ successors. Each successor uses a different queue.
2.  The Automation Engine tries to kick off both successors with a single command. 
3.  The order is random, but each of these will generate a queue_check (the FIFO query you see) - if there are multiple running in parallel, this is what causes the deadlock.

Our recommendation is that you upgrade to a ONE Automation version at or above the versions referenced in the Fix version(s) field below should this issue be encountered.

Cause

Cause type:
Defect
Root Cause: Refer to description

Environment

Release: AUTWAB99000-11.0-Automic Workload Automation-Base Edition
Component:

Resolution

This field was added on 30/03/2017. This article has not been updated yet. Refer to the "Description" or "Workaround" sections for solution information.

Fix Version(s):
Automation Engine 11.2.1 - Already released.
Automation Engine 11.1.3 - Already released.
Automation Engine 10.0.7 - Already released.

Additional Information

Workaround :
Use a single successor.