DX APM Infrastructure Agent Generates False Alerts for Azure Metrics With 100% Availability.

Products

• DX Operational Observability

Issue/Introduction

Users experience an issue where the DX APM Infrastructure Agent generates false "Danger" alerts when monitoring Microsoft Azure metrics.

Specific conditions of the issue include:

A threshold is configured to generate an alert only when a specific metric drops below 100%.

The metric is configured to be evaluated every 1 minute.

Danger alerts are triggered continuously or intermittently.

However, when reviewing the historical chart for the metric in question, the value remains steadily at 100% for the entire visible period, with no visible dips below the 100% threshold.

Cause

This issue is a result of performance issues caused by API throttling.

Because the metric is being evaluated on an aggressive 1-minute interval, temporary throttling delays the retrieval or processing of the data point for that specific minute. The alerting engine interprets this transient gap or delay as a drop in the metric value, triggering a false alert. Meanwhile, the historical chart aggregates and smooths the data, showing the continuous 100% availability once the data is eventually recorded.

Resolution

To resolve this issue and reduce alert noise caused by transient throttling, you must adjust the alerting configuration to be more tolerant of brief data delays.

Steps to resolve:

1. Navigate to the alert configuration for the affected Azure metric.

2. Locate the Period Over Threshold and Observed Periods settings.

3. Change both values from 1 to a larger value (e.g., 3 or 5).

Note: Increasing these values ensures the alerting engine waits for multiple consecutive polling cycles before triggering an alarm, effectively bypassing the temporary noise introduced by Azure API throttling. Development is working on the improvements.