Background Jobs Occasionally Marked "Cancelled" Despite Completing Successfully

Products

• Clarity PPM On Premise
• Clarity FedRAMP

Issue/Introduction

A recurring background job that runs and completes successfully is nonetheless recorded as CANCELLED (job) / JOB_RUN_CANCELLED (job run). This happens within the same second as an Oracle error appearing in the scheduler log:

Exception due to DB Connection failure08006
java.sql.SQLRecoverableException: IO Error: Socket read interrupted

 

Additionally, any other scheduled job configured as "non-concurrent" with the falsely-cancelled job can become permanently stuck in WAITING status, since the cleanup process does not revisit a job run once it reaches this particular cancelled state.

Cause

DE210486

Two independent issues in the job scheduler combine to produce this behavior:

1. As part of routine internal housekeeping, the scheduler periodically checks for stale/orphaned tracking records and, as part of that check, sends a signal to the main scheduler thread to prompt it to re-check its work queue. Under certain timing conditions, if that signal arrives at the exact moment the main scheduler thread is in the middle of a database call, the Oracle JDBC driver can interpret it as a lost connection and report it as a "Socket read interrupted" (ORA-08006) error — even though there was no actual network or database problem. This is a self-inflicted, internally generated error, not a genuine connectivity issue.
   
   
2. Separately, an orphaned-job-run cleanup process can, immediately after a job is dispatched, find that the job's tracking record wasn't written yet — because the write failed due to the error described in item 1 — and mistakenly treat the job as orphaned. It cancels the job's status even though the job itself continues running in the background and completes normally. This cleanup logic has no minimum "grace period" before acting, so it can act on a tracking record that is only milliseconds old.

Because the underlying database error is self-generated rather than a real network fault, standard connectivity workarounds (e.g., increasing JDBC retry/timeout settings) do not resolve the issue.

Resolution

DE210486, in review by Engineering

Workaround: Disable feature by running below command 

• admin toggle-feature DE176444_CLEANUP_ORPHAN_JOB_RUNS 0
• Restart all the clarity services 

Additional Information

Trade-off:

• Disabling this toggle also disables its original purpose — automatically detecting and resetting a job that is genuinely stuck in PROCESSING status (e.g., if the scheduler process died mid-start with no further heartbeat). While the toggle is disabled, such stuck jobs will need to be identified and reset manually.
  
  
• This workaround does not stop the underlying spurious ORA-08006 log entries from occurring; it only prevents them from causing a false job cancellation.