Performance issues related to large Backlog count
Article ID: 88051
Updated On:
Products
CA Automic Applications Manager (AM)
Issue/Introduction
Error Message :
N/A
It is a known issue that allowing the Backlog to grow to 3,000 or more Jobs can cause performance degradation. This will result in the one or more, but not limited to the following behaviors:
Applications Manager is a 32-bit single threaded application that only has a limited number of Java threads that can be used for database/agent communication, sub-var resolution, Job scheduling/status updates, condition handling, Java Web Client refreshes, etc.
It is highly recommended to avoid any situation where thousands of Jobs are immediately scheduled into the Backlog or to allow the Backlog to grow to 3,000 or more Jobs. Doing so will use up all available Java threads causing one or more of the above symptoms.
NOTE: Keep in mind that the 3000 Job threshold is ONLY an estimate. This number can vary greatly depending on hardware, network, and database performance as well as types of Jobs in the Backlog.
N/A
It is a known issue that allowing the Backlog to grow to 3,000 or more Jobs can cause performance degradation. This will result in the one or more, but not limited to the following behaviors:
- Jobs stuck in QUEUED status
- Slow Job status changes
- Finished Jobs are delayed to drop into History
- Slowness in overall Java Web Client refresh
- Jobs not processing
- A continued increase in backlog count
- Awapi times out during startup
- Inability to launch the Java Web Client
Applications Manager is a 32-bit single threaded application that only has a limited number of Java threads that can be used for database/agent communication, sub-var resolution, Job scheduling/status updates, condition handling, Java Web Client refreshes, etc.
It is highly recommended to avoid any situation where thousands of Jobs are immediately scheduled into the Backlog or to allow the Backlog to grow to 3,000 or more Jobs. Doing so will use up all available Java threads causing one or more of the above symptoms.
NOTE: Keep in mind that the 3000 Job threshold is ONLY an estimate. This number can vary greatly depending on hardware, network, and database performance as well as types of Jobs in the Backlog.
Environment
OS Version: N/A
Cause
Limited or no available Java threads due to large backlog count
Resolution
Avoid any situation where the 1000's of Jobs are immediately scheduled into the Backlog or to allow the Backlog to grow to 3,000 or more Jobs. If an Applications Manager instance is found to be in this situation and the backlog count continues to grow, a number of actions can be taken such as:
Fix Status: No Fix
Fix Version(s):
N/A
- Deactivating high frequency Jobs
- Deleting high frequency Jobs stuck in SELF-WAIT
- Pausing Queues and releasing them slowly one by one
- Deleting Jobs that can be rescheduled to a later time
- Spreading out Job Schedules to run throughout the day instead of all within a small timeframe
- Cleaning up Jobs that are older then 1 Virtual Day.
- Re-engineering Jobs to streamline the unnecessary use conditions and notifications.
- Increasing Java memory
- Clearing out the backlog via sql (Please contact Support for sql)
Fix Status: No Fix
Fix Version(s):
N/A
Feedback
Yes
No
Powered by
