Alarms are not closed in linked system after the APM metric stops reporting

Products

• DX Application Performance Management
• DX Operational Intelligence
• DX APM SaaS
• DX OI SaaS

Issue/Introduction

We are still seeing an open alert in our problem management system for a metric that is no longer reporting in DX APM.

 

When we check in the OI Alarms console there is no open alarm for the alert so why is this not closed in our problem system that receives alarm notifications from OI via a webhook channel?

 

Environment

Release : SAAS

Component :

Cause

If a metric for which an alert is triggered in APM stops reporting then the alert status changes to 0 - unknown.

This in turn is notified to OI which will close the open alarm and send out notifications on any appropriate channel.

Because the metric is no longer reporting the severity of the alarm is now unknown 

The defined webhook channel did not include the ${status} variable in the JSON payload and was adjusting ticket status based on the ${severity} value i.e closing the ticket when the severity returned to normal

 

 

Resolution

Include the Alarm Status - ${status} variable in the JSON payload and configure the receiving system to work against that field rather than the alarm severity when determining if an alarm is still open.

example output below

 

JSON payload options are listed here

Possible values for the ${status} variable are NEW, UPDATED and CLOSED