Alert was fired despite they were in Maintenance Window

Products

Observability

Issue/Introduction

Maintenance Window was created to prevent an Alert to be fired, but it was fired though the status of the Alert shows "In Maintenance".

An Alert can display "IN MAINTENANCE" based on any active Maintenance Windows.
When creating a Maintenance Window configured with an alert tag(s), verify the Alert also has the same alert tag.
- Otherwise, not setting alert tag(s) ensure all alerts are checked for the maintenance window.

See further documentation on Creating a Maintenance Window including discussion on "Narrow Down the Scope" of a Maintenance Window.

Cause

Scope (Conditions) of the Maintenance Window did not match the series that the Alert Fired on.

Resolution

An Alert can suppress notifications for some sources while still checking or firing on others. Hovering your mouse over an Alert's "State" column will show this detail.

Understanding the order of how Maintenace Windows evaluates by reviewing the "Scope" section within its configuration.

If alert tag(s) are configured, only alerts configured with those alert tag(s) will be evaluated. See Organize Related Alerts with Tag
If no alert tag(s) are configured, then All Alerts will be evaluated.

Additional scope parameters are set within the "refined by" section.

Within the "refined by" section identify which is being used 'AND' or 'OR'.
When using "AND" all scope parameters must be true to suppress the notifications(firing) on that source.

Use-Case:

Production environment upgrade schedule.

App servers=Monday, Exchange servers=Tuesday, File servers=Wednesday.
One Alert checks all sources in prod environment for CPU/Memory spikes.
Need to suppress notifications only on the servers when they are being upgraded while continuing to notify on any other sources experiencing CPU/Memory spikes.

Solution:

Configure three Maintenance Windows, each tagged with the alert tag(s) of env=prod with additional scope "refined by" the sources and schedules.

Window 1 is "refined by" AND Sources = app-1, app-2, on Monday, 6:00AM to 5:00PM
Window 2 is "refined by" AND Sources = exchange-1, exchange-2, on Tuesday, 6:00AM to 5:00PM
Window 3 is "refined by" AND Sources = file-1, file-2, on Wednesday, 6:00AM to 5:00PM

Result:

Reviewing your Alert during upgrade you see the current state of the Alert is 'IN MAINTENANCE CHECKING' This is because Alert Notifications are being suppressed for some sources, while still being checked on others.

Additional Information

In Maintenance Series

Expanding the Alert will show details of the alert including any "In Maintenance Series" that are currently being suppressed.

Currently there is an open feature request to improve the readability of the Scope on the main Maintenance Window screen.

Alert tag(s), notice the comma ',' placement after alert tag(s). This is an understood "AND" for example: [alert tag(s)] AND [ [point tag(s)] OR [ (source) ] ]

Maintenance Windows and Snoozing Alerts

Alert States and Lifecycle: See Section: What are Alert States?

Alerts: See Section: Organize Related Alerts with Tags