Alertmanager always sends notification of critical severity to PagerDuty

Products

• Operations Manager

Issue/Introduction

Alertmanager always sends notification of critical severity to PagerDuty regardless of the severity parameter configured for pagerduty receiver in Alertmanager. For example, the pagerduty receiver is configured with severity 'error' in Alertmanager settings.

url: https://events.pagerduty.com/generic
client_url: 'https://example.com'
severity: 'error'

However, the notification, which PagerDuty receives from Alertmanager, always has severity 'critical' regardless of what severity is configured at Alertmanager.

 

Environment

Healthwatch for VMware Tanzu 2.x

 

Cause

There are two integration types for PagerDuty integrating with Prometheus. Please refer to the PagerDuty document for more details. 

The issue only happens when selecting Prometheus as the integration type, which does not support the severity field.

 

Resolution

Select Events API v2 as the integration type in PagerDuty and configure Alertmanager properly as stated in the document. 

Enter your PagerDuty integration key in one of the following fields:

• If you selected Events API v2 as your integration type in PagerDuty, enter your PagerDuty integration key in Routing key.