Scheduler for VMware TAS [VMs] failing to trigger scheduled jobs

Products

• Operations Manager

Issue/Introduction

Scheduler app logs see this error:

ERROR 14 --- [_MisfireHandler] o.s.s.quartz.LocalDataSourceJobStore : MisfireHandler: Error handling misfires:
 Couldn't retrieve trigger: No record found for selection of Trigger with key:
 'jobs.800d3561-04f9-48ca-8af1-e01befe1243f' and statement:
 SELECT * FROM QRTZ_SIMPLE_TRIGGERS WHERE SCHED_NAME = 'scheduler' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ? 

Cause

This is a bug in Scheduler for TAS [VMs] When there is a row in QRTZ_TRIGGERS of TRIGGER_TYPE ‘SIMPLE’ that does not have a corresponding row in QRTZ_SIMPLE_TRIGGERS and there are misfired triggers then we see this behavior.

Environment

Product Version: 2.7

Resolution

The ‘SIMPLE’ jobs are added by scheduler when you perform a one-off job execution, for example using cf run-job. If you don’t care about losing data about these one-off triggers then we think it is safe to delete the inconsistent rows from QRTZ_TRIGGERS to restore service
 

Workaround

Note: Any manual modifications to the database can lead to irreversible damage. If you are not sure, please contact VMware Tanzu support.

Follow these instructions:

1. Backup the scheduler database
2. Review the rows in QRTZ_TRIGGERS that do not have matching rows in QRTZ_SIMPLE_TRIGGERS:

   • SELECT * FROM QRTZ_TRIGGERS WHERE TRIGGER_TYPE = 'SIMPLE' AND TRIGGER_NAME NOT IN (SELECT TRIGGER_NAME FROM QRTZ_SIMPLE_TRIGGERS);

3. Having reviewed the rows then you can proceed to delete them from the database:

   • DELETE FROM QRTZ_TRIGGERS WHERE TRIGGER_TYPE = 'SIMPLE' AND TRIGGER_NAME NOT IN (SELECT TRIGGER_NAME FROM QRTZ_SIMPLE_TRIGGERS);

4. Run cf job-history to confirm that the misfired triggers have been run successfully and that subsequent scheduled jobs are run appropriately (service has been restored).

Permanent Fix

VMware Tanzu Engineering team is working on a code fix to be released in future version of Scheduler for VMware TAS [VMs].