Alert fires on new source even though target window has not been met

Products

• Insights
• DX OpenExplore
• Observability

Issue/Introduction

Alerts are firing when a new source(s) report in breaching the threshold briefly but not for the full duration of the configured target window.

Cause

When an alert sees a true value proceeded by "No Data" value this is a special situation that will cause the alert to fire immediately and not wait for the target threshold.

Resolution

For new sources that breach the Alert condition only briefly, we need to include additional check similar to configuring for "No Data" scenarios.

Following the below process to add the additional checks.

1. Identify the original alert condition.  ts(<some query value>)
2. Wrap the original condition with mcount() to test if there are more then a single true data point received.  mcount(5m, ts(<some query value>)) > 3
   Note: Test values should be based on your specific data patterns.
3. Use the AND operator to combine your test condition with the condition you want to be alerted on.  mcount(5m, ts(<some query value>)) >3 AND ts(<some query value>)
   
4. Results, both sides of the AND statement must be true before your alert will fire.

Additional Information

mcount Function

Alerting on Missing Data in Individual Time Series