Delays processing RUNNING events for box jobs

Products

• Autosys Workload Automation

Issue/Introduction

Delays are observed in the processing of RUNNING events for box jobs...

RUNNING         11/22/2022 10:20:12    1  PD  11/22/2022 10:21:01

 

 

Environment

Release : 12.0/12.1

Cause

Processing a RUNNING event for a box job involves evaluating the jobs inside the box to see which ones can start immediately versus which ones will need to be placed into the ACTIVATED state. A subset of scheduler threads are designated to handle these evaluations. There is a limit imposed on the number of these box evaluation threads available.
This problem can occur when there are many box jobs attempting to start concurrently. Some may have to wait for an available box evaluation thread when there are not enough available to handle all of the boxes attempting to start. To determine whether this is the root cause for the delays, here are a couple of other things to check...

1. The RUNNING events for box jobs are the only events experiencing the delays. Other events are processing quickly.
2. Put the Scheduler in debug mode with the following ISDBGACTIV setting in the config.$AUTOSERV file...

ISDBGACTIV=LIGHT,HEAVY

Let the scheduler run in this debug mode when there are many box jobs attempting to start concurrently. Look for the following type of message in the event_demon log with debug enabled...

[11/22/2022 10:20:01.8355] 2719385 139743166953216 L:2 bxx.cpp 103 EHX[139743166953216]->BXX. Hit BXX Thread Ceiling at 5

Resolution

The max number of box evaluation threads allowed is equal to the SCHED_SCALE setting for the scheduler. By default, this is set to 5. To increase the maximum number of box evaluation threads, you can increase the SCHED_SCALE setting. Steps for this can be found here...

https://techdocs.broadcom.com/us/en/ca-enterprise-software/intelligent-automation/autosys-workload-automation/12-1/administrating/ae-administration/define-tuning-parameters-for-the-scheduler.html