DX O2 APM Alerts triggered at incorrect Threshold

Products

• DX SaaS
• DX Operational Observability

Issue/Introduction

we see the alerts were triggered for threshold of 81, while the definitions is set for 90 .

Environment

DX O2

Resolution

The discrepancy and confusion was caused by a user changed of the thresholds 

By looking at elastic search, we found Alarms with the same alert_external_id emitted on previous days earlier has the danger and caution threshold set to that correspond to the Alarm, in this example 80, that explain why the alarm was triggered.

The Elastic "itoa_alarms_all" index, confirms the threshold changes:

                      {
                            "propName": "danger_threshold",
                            "oldValue": [
                                "80"
                            ],
                            "newValue": [
                                "90"
                            ]
                        }