Resolving Alert Flapping Caused by Ingestion Delays & Query Misconfigurations

Products

• DX OpenExplore

Issue/Introduction

Alerts in Wavefront are continuously firing and resolving (flapping) over a period where the underlying metric condition remains continuously in a breach state.

Symptoms

• Alerts trigger, auto-resolve, and re-trigger within short intervals (example, 1–2 minute gaps).

• Alert history displays multiple SEVERE firing events within a window where the metric was expected to be continuously breaching.

• Interactive charts show complete, continuous data after the fact due to backfilling, but real-time alert evaluation sees missing or incomplete points.

Environment

Wavefront (VMware Aria Operations for Applications)

Cause

Alert flapping in these scenarios is typically caused by a combination of data ingestion delays and query configuration issues:

1. Ingestion Delay: Raw metrics (such as AWS CloudWatch data) often arrive with a delay of 2 to 5 minutes. During real-time query evaluation, the most recent time buckets contain no data points yet.

2. Incorrect Query Logic: Using functions like moving averages (mavg) over windowed conditions (e.g., mavg across 15 minutes combined with an if condition) without accounting for ingestion lag causes the query to evaluate incomplete or missing data buckets as FALSE or NO DATA. This causes Wavefront to prematurely resolve the alert before backfilled data arrives, leading to continuous fire/resolve cycles.

Resolution

1. Shift the Evaluation Window Using lag()

The primary and most effective solution to handle metrics with consistent ingestion delays is to shift the evaluation window back using the lag() function.

• How it works: The lag() function instructs Wavefront to evaluate the query condition against data as it existed $N$ minutes ago. This gives the pipeline time to backfill incoming points so the alert engine always evaluates a complete dataset.

• Example Update:

  • Original / Problematic Query:

    if(mavg(15m, ts(aws.metric.name)) > 80, ts(aws.metric.name) > 80) (or similar misconfigured conditional mavg logic)

  • Updated Query (with 2-minute buffer):

    lag(2m, ts(aws.metric.name) > 80)

Note: A 2-minute lag buffer ensures that Wavefront checks data from 2 minutes ago, preventing temporary resolution drops while maintaining near real-time operational visibility.

2. Alternative UI Configuration Solutions

If applying a lag() buffer to the query is not desired (e.g., when immediate initial firing is required for manual operational response):

• Adjust the Resolve Window (Resolve Delay):

  Configure the Resolve Window (or Minutes to Resolve) in the alert UI settings to 5–10 minutes. This ensures Wavefront requires the condition to evaluate as false continuously for 5–10 minutes before auto-resolving, preventing a temporary 2–5 minute AWS data gap from closing the alert.

• Use the default() Function:

  Wrap the query with default() to hold the last reported state during ingestion gaps:

  default(5m, ts(aws.metric.name)) > 80

  This retains the last known value for up to 5 minutes during data gaps without delaying initial trigger notifications.

Verification & Testing

To verify the fix:

1. Apply the updated lag() query or modified resolve window to a test alert.

2. Monitor the alert history during a known metric breach period.

3. Confirm that the alert remains in a continuous FIRING state without intermittent resolution entries during the breach window.