Time Slicing job fails with ORA-00060: deadlock detected while waiting for resource

Products

• Clarity PPM On Premise

Issue/Introduction

The Time Slicing job fails intermittently with a deadlock error in the BG logs.

MSSQL:
Transaction (Process ID) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

Oracle:com.niku.union.persistence.PersistenceDeadlockException:
SQL error code: 60
Error message: [CA Clarity][Oracle JDBC Driver][Oracle]ORA-00060: deadlock detected while waiting for resource

Note: The failure is not fatal, as the job will recover and process the records the next time it runs.

Cause

This can be due one or more of the following causes:

1. Large amount of XOG activity was introduced while the Time Slicing job is running.
2. A custom process or job is running at the same time as the Time Slicing job.
3. A custom process or trigger was placed on an action that initiates the Time Slicing job.

Resolution

Solution 1: Use the following page (Administration > Data Administration > Reports and Jobs > Time Slicing job > Incompatible Jobs tab)
to check or edit the Time Slicing job incompatibility list:

• The following jobs should be set per best practices:
• Datamart Extraction (Default)
• Delete Investments (Default)
• Datamart Rollup (Default)
• Rate Matrix Extraction (Recommended)
• Post Timesheets (Recommended)
• Post Transactions to Financials (Recommended)
• Investment Allocation (Recommended)
• Import Financial Actuals (Recommended)
• Update Earned Value and Cost Totals (Recommended)
• Load Data Warehouse (Recommended) 

Any job in the list will wait for the Time Slicing job to finish before it is launched.

The following query can also be used:
select
I.JOB_DEFINITION_ID,
I.NON_CONCUR_JOB_DEF_ID,
k.job_code
FROM
CMN_SCH_JOB_NONCONCUR I,
CMN_SCH_JOB_DEFINITIONS J,
CMN_SCH_JOB_DEFINITIONS K
Where I.Job_Definition_Id = J.Id
and I.NON_CONCUR_JOB_DEF_ID = K.id
and J.JOB_CODE in ('Time_Slicing')
order by J.JOB_CODE

Solution 2:
If there is a long-running process/XOG activity, pause the Waiting/Scheduled Time Slicing job.

Obtain the results of:
select * from PRJ_BLB_SLICEREQUESTS order by id

Solution 3:
Set the Time Slicing job from running every minute to every 5 minutes. This will generate the same data without impact.
This theoretically would alleviate the contention considerably as the Time Slicing is not forced to process bulk data change, e.g. a XOG or process that reads/updates data such as allocation.

Solution 4: For MSSQL databases, reach out to the DBA team to see what configuration can be set to view the capture and view the deadlock graphs.
https://support.microsoft.com/en-us/help/224453/inf-understanding-and-resolving-sql-server-blocking-problems 

Solution 5: If a custom trigger is used, disable the trigger and check if the issue happens again.
Have the DBA team review the DDL of the custom trigger.

Additional Information

Clarity PPM: How Should the jobs be scheduled?