Alert not firing due to data lag

Products

• Observability
• DX OpenExplore

Issue/Introduction

Alert should be triggered when we see any errors. "Test Condition" to confirm that the alert should have been triggered. 

 

Resolution

An alert is evaluated only on the data that is available in that moment and not on backfilled data. Reviewing the data pattern in live view confirms the "typical" data shape.

To understand the data shape, view your Alert Condition query in a Live Chart View, setting the time window to 5 or 10 minuets.

For example,

• Chart View:  Live View.
• Chart Time Window: 10 Minuets
• Chart Type: Point Plot

 

 

The example above shows a delay of data being ingested. 

Identify how long the data points typically take to arrive and adjust your Alert Condition functions accordingly to account for any delays.

 

Please note there are multiple methods of accounting for this situation, most often used function is the lag function.

lag(3m, ts(metricName)

 

Please review our documentation Wavefront Query Language Reference for other Standard or Moving Window Time functions that could also fit your Alert requirements or reach out to Support@Broadcom  for further assistance. 

Additional Information

lag Function - Returns earlier data values from the time series described by the expression.

msum Function - Returns the moving sum of each time series over the specified time window.