Jobs stuck in generating status due to WP exhaustion - Automic Automation

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

Jobs remain in Generating status for extended periods. Performance monitoring shows high process utilization for Work Processes (WPs), specifically B01, B10, and B60 processes are maxed out at 100%.

 

Environment

• Product: Automic Automation Engine
• Version: 24.x and higher
• Database: Oracle

Cause

There are two primary factors contributing to this issue:

1. Workload Exhaustion: The current volume of jobs exceeds the capacity of the active Work Processes (WPs).
2. Configuration Mismatch: The WP_MIN_NUMBER setting in UC_SYSTEM_SETTINGS is configured lower than the number of physical processes running. In Automic, any running server processes beyond the WP_MIN_NUMBER threshold are automatically converted into Dialog Processes (DWPs). If this setting is too low, the system may lack enough WPs to handle job generation, even if many processes are technically "running."

Resolution

⚠️ IMPORTANT: Always test configuration updates first in a non-production environment. Take a backup of the UC_SYSTEM_SETTINGS variable before implementing.

Ensure that the minimum number of Work Processes is configured to reflect the actual number of WPs expected on each node.

1. Log into Client 0.
2. Open the UC_SYSTEM_SETTINGS variable.
3. Locate the WP_MIN_NUMBER key.
4. Evaluate the value against the number of server processes currently running on your nodes.
5. Increase the WP_MIN_NUMBER to ensure your running processes remain in 'WP' mode rather than being converted to 'DWP'. For example, if you have 15 processes running and the setting is 6, 9 of them will become DWPs. To keep 12 of them as WPs, update the setting to 12.
6. Save the changes. A restart is not required for this change to take effect.

Verify the process roles in the Administration perspective under Processes and Utilization. Processes should now appear as 'WP' in accordance with your new setting.

Additional Information

High record counts in the EH (Execution History) table can further contribute to performance degradation during job generation. It is recommended to maintain a tidy EH table through regular cleanup utilities.