Increased volume of alerts sent from RESTMon to OI

Products

• DX OI SaaS

Issue/Introduction

Our tenant receives alerts from 4 RESTMon instances, and until 29-Mar (around 4pm UTC) the volume of alerts was normal - less than 1000 per day, however after 29-Mar this number increased dramatically to around 4 million per day. The sources of the alerts have not changed, can you please let us know if anything in the backend changed to cause this increase?

According to Product Engineering, it seems like all INFO-level alerts are being sent to OI while previously this was not happening.

 

Cause

A change was made to ingest a closed alarm as well to address some of the use cases. It accidentally created a new closed alarm if the alertID is not changing. 

Resolution

Previous change undone via a new version of RESTMON (dx-restmon:2.2.5 is available with the fix.)