RaaS scheduled jobs trigger unexpectedly after service restart

Products

• VCF Automation
• VMware Salt

Issue/Introduction

• RaaS services are stopped for a period of time (e.g., during a network outage or maintenance window).
• Upon restarting RaaS services, all jobs that were scheduled to run during the downtime execute immediately and simultaneously.
• This "burst" of activity can lead to unplanned production changes, reboots, or resource exhaustion on Salt Masters and Minions.

Environment

• VMware Aria Automation Config (formerly SaltStack Config)
• RaaS versions prior to 8.17.0

Cause

In legacy versions of RaaS, the scheduler pre-calculates future scheduled jobs and stores them in the database with a status of created. When the RaaS service starts, the scheduler queries the database for all outstanding "futures" where the scheduled start time is in the past (start_time <= now).

Because these older versions lacked an age-validation mechanism, the scheduler queued and dispatched every missed job to background workers immediately upon startup, regardless of how much time had elapsed since the original scheduled start time.

Resolution

This issue is resolved in RaaS version 8.17.x and higher through the implementation of a delay-threshold mechanism.

The updated scheduler now includes logic to automatically discard outdated jobs:

1. Delay Threshold: A configurable parameter, scheduler_max_job_delay_cycles (default: 100), is used in conjunction with the schedule_cycle (default: 10 seconds) to define a maximum allowable delay (approx. 16.6 minutes by default).
2. Cutoff Evaluation: Upon startup, RaaS calculates a skip_cutoff timestamp: skip_cutoff = now - (max_delay_cycles * schedule_cycle)
3. Automatic Skipping: Before a job is queued, its start_time is compared against the skip_cutoff. If the job is older than the cutoff, it is marked as Skipped and logged as "outdated" instead of being dispatched for execution.

Additional Information

If an upgrade is not immediately possible, use the following steps to prevent unexpected job execution after a planned outage:

1. Disable Schedules: Before stopping RaaS services, manually disable any critical or high-impact job schedules.
2. Database Maintenance: (Advanced) Before restarting RaaS services, a database administrator can manually update the status of pending jobs in the schedule table of the RaaS PostgreSQL database to prevent them from firing.
3. Post-Restart: Re-enable the schedules once the RaaS services are stable and the missed window has passed.