AutoSys Agent Hangs or Autoping Times Out with "acquired initiator of class CONTROLCLASS. 0 left" Under High Concurrent Job Volumes

Products

Autosys Workload Automation

Issue/Introduction

After migrating or running the Workload Automation System Agent on Red Hat Enterprise Linux (RHEL) 8, autoping commands intermittently or permanently time out on the scheduler side. Concurrently, a high burst of simultaneous jobs causes the agent to stop responding, leaving workloads stuck in a STARTING or RUNNING state.

Review of the agent's initiatormanager.log reveals the following patterns:

Core Mailbox.outbound.CybInitiatorManager.executeWOB[:703] - [...]/MAIN acquired initiator of class CONTROLCLASS. 0 left
Core Mailbox.inbound.CybInitiatorManager.executeWOB[:731] - |[...] CONTROL STATUS Port(...) Address(...) has been put to waiting queue

Environment

Product: AutoSys Workload Automation AE (v12.x)
Component: Workload Automation System Agent (v12.1)
Operating System: Red Hat Enterprise Linux (RHEL) 8.x (often observed following a migration from RHEL 7)

Cause

The failure stems from an operating system-level bottleneck related to Inter-Process Communication (IPC) resource allocation rather than an application bug.

OS Message Queue Saturation: In modern RHEL 8 environments, the default maximum size for a System V IPC message queue (kernel.msgmnb) is configured to 16 KB (16,384 bytes).
High Concurrency Triggers Saturated State: When a high burst of concurrent jobs or intense testing cycles (e.g., executing hundreds of simultaneous requests or multiple autoping -S connections) hits the agent, the internal dynamically managed communication queue fills up rapidly. If an individual Automated Framework Message (AFM) averages 1 KB, the default queue can only sustain 16 to 64 concurrent messages before reaching its capacity threshold.
Undefined Code Block State: Running command lookups via ipcs -a confirmed the agent queue utilizing 16,243 bytes out of 16,384 bytes (93 messages total). Once the space is completely exhausted, the underlying libc C-runtime named pipe logic enters an undefined block state. This prevents pre-existing messages from moving forward or clearing out, permanently blocking communication between the cybAgent binary and its execution process spawners

Resolution

To support heavy concurrency thresholds safely (scaling past 500+ simultaneous requests), the Linux kernel parameters must be adjusted to explicitly allocate 64 KB of queue space.

Step 1: Increase the Kernel Message Queue Limit

Log in as a root-level user or a user with elevated sudo privileges and complete the following commands:

Open the system's kernel control configuration file using a text editor:
Bash
```
sudo vi /etc/sysctl.conf
```
Append the following parameter limit configuration line to the bottom of the file:
Plaintext
```
kernel.msgmnb = 65535
```
Save the changes and exit the file.

Step 2: Dynamically Apply the Configuration

Reload the kernel configurations to apply the changes instantly without triggering an entire operating system reboot:

Bash 
sudo sysctl -p

Step 3: Perform a Agent Restart

Important Recovery Note: If the agent has been stuck in a hung state with a large backlog of old, unresolved production events, directly restarting the component may cause it to unexpectedly dump and re-trigger those historical stale processes back into your environment simultaneously.

To perform a clean recovery, it is recommended to conduct a cold start by clearing out historical records:

Stop the System Agent service safely.
Navigate to the agent installation path and locate the database directory.
Take a backup by renaming or moving the directory (e.g., rename database to database_old).
Start the System Agent service. It will dynamically spin up a clean database folder and initialize with the newly scaled 64 KB message space boundary.

Important Note: Cold start will eliminate the current status of the jobs and will start triggering for the next start.