AAI Alerts Delivered ~60 Minutes Late Due to JMS Backlog

Products

• Automation Analytics & Intelligence

Issue/Introduction

Jobstream alerts (Exceeded Deadline, Late Start, Started, Complete, etc.) are consistently delivered to end users approximately 60 minutes after the actual event occurred.

Additionally, the alert content may appear to contain contradictory information. For example, an alert may state that a Jobstream was "28 minutes late" even if the total "Elapsed Time" for the run was only 20 minutes.

Environment

Product: Automation Analytics & Intelligence (AAI)
Version: 24.4.0

Cause

The issue is caused by a sustained JMS message backlog on the JobStreamRunAlerts topic.

In version 24.4.0, the AlertService consumer on this topic is single-threaded by design. Each message requires multiple synchronous database queries to validate alert activations, capping throughput at approximately 130–140 messages per minute. In environments with high event volume (often exacerbated by the "Multiple SLAs per Day" feature), the publication rate can exceed this capacity by 3x or more.

The backlog stabilizes at the publishableAlertTtl boundary (default 60 minutes). This means:

1. Alerts sit in a queue for 60 minutes before the consumer can process them.
2. The "Late" calculation in the alert is based on the time the message was processed, not when the event occurred, leading to the mathematical inconsistencies observed.
3. Approximately 60–70% of alerts may be silently discarded by the broker if they expire before they can be processed.

Resolution

This issue is addressed in AAI version 26.0 (formerly 24.5.0), which introduces concurrent JMS consumers and optimized database queries to significantly increase throughput and eliminate the backlog.

Workaround for Version 24.4.0

To reduce the delivery delay immediately, you can lower the expiration threshold for alerts. This caps the maximum lag but may result in a higher rate of "missed" alerts (silent discards).

1. Stop the AAI Service.
2. Launch the AAI Configuration Tool.
3. Navigate to Server Parameters > Advanced Parameters.
4. Locate the parameter: publishableAlertTtl (Default value is 60).
5. Change the value to 15 or 30 (minutes).
6. Save the settings and Restart AAI.

Note: A minimum TTL of 15 minutes is recommended to ensure the system has enough time to process alerts during normal Jammer cycles.