Alert didn't fire even though the condition was true
search cancel

Alert didn't fire even though the condition was true

book

Article ID: 406615

calendar_today

Updated On:

Products

Observability DX OpenExplore

Issue/Introduction

The condition was true for 1 min trigger window.  Why is the live view then showing that the query never breaches zero in this case?

Depending on when the alert is checked a condition of one true and one false value within trigger window results in the alert not firing.

For example:

  • Your chart shows the condition turns true at 3:53:48 and returns to false at 3:55:47 - This is seen as true for two full minutes.
  • However, in reality the condition was checked at 3:53:45 (false), at 3:54:50 (true), at 3:55:50 (false)

This third check equaling false means the alert was not true for 1-minute and does not fire.

Resolution

An alert is triggered when its defined condition results to at least one true value and zero false values within the specified "Trigger Window".

How to build a successful Alert Condition first requires understanding of your data patterns.

To understand patterns of data points, examine the points in close detail as they arrive using a test chart of your Alert Condition.

    • Chart Settings: Live [5M 5-Minute]
    • Chart Type: Point Plot.
    • Query: Use a time series without additional functions that may mask data patterns.

Patterns to look for to understand impacts to your alert conditions.

    • Are points arriving consistently or inconsistently.
    • Are metrics used in the Alert Condition arriving at the same time, or do they arrive at different times. 
    • Are there multiple points per minuet?

Next understand the lifecycle of when an alert is checked and the granularity of the checking process ensures building successful Alerts. For further discussion on this review Article Alert States and Lifecycle.

Additional Information