Jobs stuck in Waiting, Scheduled, or Processing in Clarity

Products

• Clarity PPM On Premise
• Clarity PPM SaaS

Issue/Introduction

The job scheduler appears to have halted and it will not process any jobs in the queue. The jobs are in Waiting/Scheduled/Processing/Cancelled status but not changing statuses.

Jobs are taking more time than normal to complete. The job has been shown in processing status for a long period of time, how do we fix this error/issue/performance issue?

Examples of jobs that can be impacted:

Create and Update Jaspersoft Users job Datamart Rollup job Index contents and documents for searches job Tomcat access log import/analyze job Synchronize Jaspersoft roles job Time Slicing Sync job

Send Action Item Reminders job Update Aggregated Data job Load Data Warehouse (DWH) job (Incremental or full) Time Slicing job Datamart Extraction job

Other jobs of different types may or may not be running. 

Cause

This is caused by a disruption in the job scheduler, which runs on the background (BG) service. If the jobs were not paused and/or were not allowed to complete prior to the disruption, the job scheduler will not recover.

• Any action such as the below will affect the Clarity job scheduler running on the BG service.
  • Server upgrades
  • Database refreshes
  • Outages
  • Connection issues
  • Patch installs will 
• There is a chance that after a release update the jobs that were in flight (i.e. in the Processing status) will be orphaned. Therefore, they will appear as never completed. This can then force your Clarity administrator to cancel, delete, and re-enter the job. In some instances, scheduled jobs may no longer run.
• SaaS customers: As the SaaS team will not be performing the actions to reschedule or re-enter jobs in the event the issue occurs, it is highly recommended to pause all jobs in the Waiting and Scheduled states prior to the release update activity.
• On-Premise customers: Check with your internal IT team for any maintenance performed recently that could have caused a disruption to the BG service.

Resolution

 

Step 1: If a job is stuck in processing at DB level, but not in the UI

See KB: Single Clarity Job is stuck in Processing State

If the above does not help, proceed through the following steps.

Step 2: Check for issues in 'Scheduled Jobs'

Check for issues via Home > Reports and Jobs > Jobs > Scheduled Jobs

Important: Run these steps in sequential order

1. Take the load off the job scheduler

1. Filter for Job Status = 'Waiting'
2. Select the 'Waiting' jobs and click the 'Pause' button.
3. Filter for Job Status = 'Scheduled'
4. Select the 'Scheduled' jobs and click the 'Pause' button.
5. Filter for Job Status = 'Cancelled'
6. Make a note of the schedule, as the job will need to be re-entered later. A job in the 'Cancelled' status cannot be resumed, therefore will need to be deleted.\Delete all the 'Cancelled' instances of the following jobs:

• • Time Slicing
  • Post Timesheets
  • Rate Matrix Extraction
  • Investment Allocation
  • Datamart Extraction/Datamart Rollup
  • Load DWH
  • Refresh Data Warehouse OData Model (for SaaS customers)

2 - Cancel all jobs in 'Processing' state

1. Filter for Job Status = 'Processing'
2. Cancel the job(s).

3 - Run the Clean User Sessions job

1. Run this job in immediate mode to tell if the job scheduler is working correctly. The status should go to 'Processing'.
   1. If it does go to the 'Processing' state, allow the job time to complete. It should take less than 5 minutes to complete.
   2. If the job does not go into the 'Processing' state, contact Support for analysis.
2. Once the immediate jobs are running:
   1. Select all the 'PAUSED' jobs and set them to 'Resume' allowing the jobs to get back into the normal schedule.
   2. Re-enter the previously cancelled jobs in Step 1.
3. If the issue still persists, contact Support to implement steps in the following article: Scheduled Jobs Do Not Get Processed by the Job Scheduler

Notes

1. If job runtime has increased for one or more jobs after migration to a different database, the DBA team will need to perform analysis to get the runtime down to where it was prior to the migration.
2. If a job fails, a subsequent run of the job should result in completion. By design, the data is rolled back so that prior data is intact.
   
   TIP: Ahead of an upgrade or patch install, the job scheduler should be in a no-load state, which will prevent orphan jobs, database sequencing issues, and out-of-sync job starts.
3. After the planned outage completes, the following can be performed on the Reports and Jobs > Scheduled page, after the BG service(s) are started.
   1. Run an immediate Time Slicing job to ensure it completes.
   2. Check the Administration > Time Slices page to ensure the 'Last Run' date reflects the most current time.
   3. Checkmark all PAUSED jobs and click the RESUME button. The jobs will be auto-scheduled.
4. These steps will ensure a smooth job scheduler startup as it will prevent jobs from getting stuck in the 'Processing', 'Waiting', and 'Scheduled' states.
5. If a job is in any of the following statuses and set to Cancelled, the job instance scheduled has to be rescheduled, as a cancelled job cannot be resumed.
   • Scheduled
   • Waiting
   • Processing

Additional Information

Note: OP customers, in some rare cases may have unchecked Run Scheduler in CSA for BG service. Make sure you have not done this

See also:

• Scheduled Jobs Not Processed by the Job Scheduler (BG Service Issue)