Jobs Stuck in Waiting, Scheduled, or Processing in Clarity

book

Article ID: 3995

calendar_today

Updated On:

Products

Clarity PPM On Premise Clarity PPM SaaS

Issue/Introduction

The job scheduler will appear to have halted and it will not process any jobs in the queue. The jobs are in Waiting/Scheduled/Processing/Cancelled status but not changing statuses.

Cause

  • This is caused by a disruption in the job scheduler which runs on the background (BG) service. If the jobs were not paused and/or were not allowed to complete, prior to the disruption, the job scheduler will not recover.
  • Any action such as the below will affect the Clarity job scheduler running on the BG service.
    • Server upgrades
    • Database refreshes
    • Outages
    • Connection issues
    • Patch installs will 
  • There is a chance that after a release update, the jobs that were in flight, i.e. in the Processing status will be orphaned, therefore will appear as never completing. This can then force the Clarity administrator on the customer side to cancel, delete, and re-enter the job. In some instances, scheduled jobs may no longer run.
  • For SaaS customers: As the SaaS team will not be performing the actions to reschedule or re-enter jobs in the event the issue occurs, it is highly recommended to pause all jobs in the Waiting and Scheduled states prior to the release update activity.
  • For On-Premise customers: Please check with your internal IT team for any maintenance performed recently that could have cause a disruption to the BG service.

Resolution

First make sure the Option under the NSA Background  Tab 'Run Job Scheduler' is checked. 

If it is not please check that box.  This should then resolve the issue.

Easy way to find out if a job is stuck in processing at DB level, but not in the UI:

Single Clarity Job is Stuck in Processing State

If the above did not help, proceed through the following steps.

 

Please perform the steps in sequential order in Home >  Reports and Jobs > Jobs > Scheduled Jobs

 

Step 1 - Take the load off the job scheduler

  1. Filter for Job Status = 'Waiting'
  2. Select the 'Waiting' jobs and click the 'Pause' button.
  3. Filter for Job Status = 'Scheduled'
  4. Select the 'Scheduled' jobs and click the 'Pause' button.
  5. Filter for Job Status = 'Cancelled'
  6. Make a note of the schedule, as the job will need to be re-entered at a later time. A job in the 'Cancelled' status cannot be resumed, therefore will need to be deleted.\Delete all the 'Cancelled' instances of the following jobs:
  • Time Slicing
  • Post Timesheets
  • Rate Matrix Extraction
  • Investment Allocation
  • Datamart Extraction/Datamart Rollup
  • Load DWH
  • Refresh Data Warehouse OData Model (for SaaS customers)

Step 2 - Cancel all jobs in the 'Processing' state.

  1. Filter for Job Status = 'Processing'
  2. Cancel the job(s).

Step 3 - Run an immediate instance of the Clean User Sessions job.

  • This is to tell if the job scheduler is working correctly.
  • The status should go to 'Processing'.
    • If it does go to the 'Processing' state, allow the job time to complete. It should take less than 5 minutes to complete.
    • If the job does not go into the 'Processing' state, please contact Support for an analysis.
  • Once the immediate jobs are running:
  1. Select all the 'PAUSED' jobs and set them to 'Resume' to let it get back to the normal schedule.
  2. Re-enter the previously cancelled jobs in Step 1.

Notes:

  • If job runtime has increased for one or jobs after migration to a different database, the DBA team will need to perform analysis to get the runtime down to where it was prior to the migration.
  • If a job fails, a subsequent run of the job should result in a completion. By design, the data is rolled back so that prior data is intact.

    TIP: Ahead of an upgrade or patch install, the job scheduler should be in a no-load state which will prevent orphan jobs, database sequencing issues, and out-of-sync job starts.
  • After the planned outage completes, the following can be performed on the Reports and Jobs > Scheduled page, after the BG service(s) are started.
    • Run an immediate Time Slicing job to ensure it completes.
    • Check the Administration > Time Slices page to ensure the 'Last Run' date reflects the most current time.
    • Checkmark all PAUSED jobs and click the RESUME button. The jobs will be autoscheduled.
  • These steps will ensure a smooth job scheduler startup as it will prevent jobs from getting stuck in the 'Processing', 'Waiting', and 'Scheduled' states.
  • If a job that is in any of the following status: Scheduled, Waiting, Processing and set to Cancelled, the job instance scheduled effectively has to be rescheduled, as a cancelled job cannot be resumed.

Additional Information

See also Scheduled Jobs Do Not Get Processed by the Job Scheduler