DX OI - Alarm explosion caused by the APM integration - How to delete ALL or many APM alarms at once?

Products

• DX Operational Intelligence

Issue/Introduction

APM integration is producing too many alarms causing many incidents to be created in ServiceNow

1) How to delete all or many APM alarms at once?

2) How to disable the creation of alarms in OI?

3) How to quickly disable the APM 10.7 to DX OI integration?

 

Environment

DX Operational Intelligence 1.3.x, 20.2.x onwards

Resolution

1) How to delete all or many APM alarms at once?

STEP 1: Find out the tenant name

REST Endpoint:

POST {Elastic-Endpoint}/ao_tenants_1_1/_search?size=200&pretty

Example:

http://es.munqa001493.abc.net/ao_tenants_1_1/_search?size=200&pretty

In below example: DOI132-USERSTORE

 

STEP 2: Delete Alarms

Option a) Delete a specific alarm

1) Find the alarm you want to delete and collect the "Alarm ID" :

- From DX OI Alarm Console 

- Or by querying elastic:

Endpoint: 

{Elastic-Endpoint}/ao_itoa_alarms_all_1*/_search

Example:

http://es.lvntest010772.abc.net/ao_itoa_alarms_all_1*/_search

Body Template:

{
    "query": {
        "bool": {
            "must": [
               {
                   "match": {
                      "doc_type_id": "itoa_alarms_apm"
                   }
               },{
                   "match": {
                      "@tenant_id": "<tenant_id>"
                   }
               }
            ]
        }
    }
}

2) Delete the alarm(s):

REST Endpoint:

POST {Elastic-Endpoint}/ao_itoa_alarms_all_1*/_delete_by_query

Example:

http://es.lvntest010772.bpc.broadcom.net/ao_itoa_alarms_all_1*/_delete_by_query

Body template:

{
    "query": {
        "bool": {
            "must": [
               {
                   "match": {
                      "doc_type_id": "itoa_alarms_apm"
                   }
               },{
                   "match": {
                      "alarm_unique_id": "<alarm-id>"
                   }
               },{
                   "match": {
                      "@tenant_id": "<tenant_id>"
                   }
               }
            ]
        }
    }
}

For example:

{
    "query": {
        "bool": {
            "must": [
               {
                   "match": {
                      "doc_type_id": "itoa_alarms_apm"
                   }
               },{
                   "match": {
                      "alarm_unique_id": "ibntest003751-7820caea-2cae-42a2-8028-ab5199328e9f-ibntest003751"
                   }
               },{
                   "match": {
                      "@tenant_id": "DOI132-USERSTORE"
                   }
               }
            ]
        }
    }
}

 

Option b) Delete all APM alarms

REST Endpoint:

POST {Elastic-Endpoint}/ao_itoa_alarms_all_1*/_delete_by_query

Example:

http://es.lvntest010772.bpc.broadcom.net/ao_itoa_alarms_all_1*/_delete_by_query

Body template:

{
    "query": {
        "bool": {
            "must": [
               {
                   "match": {
                      "doc_type_id": "itoa_alarms_apm"
                   }
               },{
                   "match": {
                      "@tenant_id": "<tenant_id>"
                   }
               }
            ]
        }
    }
}

For Example:

{
    "query": {
        "bool": {
            "must": [
               {
                   "match": {
                      "doc_type_id": "itoa_alarms_apm"
                   }
               },{
                   "match": {
                      "@tenant_id": "DOI132-USERSTORE"
                   }
               }
            ]
        }
    }
}

 

2) How to disable the creation of alarms in OI?

Option #1) 

a) Disable the "propagate to teamcenter" option 

b) Disable the alarm itself using the "Activate" checkbox

Option #2) disable the APM integration

 

3) How to quickly disable the APM 10.7 to DX OI integration?

Set introscope.enterprisemanager.doi.enabled=false in the MOM properties file.

This is hot property and change will be taken into account after 60 seconds

 

Here is an example of the EM log file confirming that OI integration has been disabled:

 

11/20/20 05:36:05.860 PM IST [VERBOSE] [PO Async Executor] [Manager.PropertyService] Changed introscope.enterprisemanager.doi.enabled=false (true)
11/20/20 05:36:05.862 PM IST [INFO] [PO Async Executor] [Manager.APM.OI] APM OI disabled

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html