AIOps - jarvis_jmetrics_1.0 growth up continuously

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

We have noticed that jarvis_metrics_1.0 index grew up to 71 GB. Is this normal?

Environment

DX Platform 21.3.1

Resolution

If the index name is ""jarvis_jmetrics_1.0_1"" :  open a support case to investigate the root cause.

 

If the index name is "jarvis_jmetrics_1.0" it is incorrect:

 

Reason:

- it can happen, when the ingestion to index, happens before the index creation from jarvis_api. And it won't be eligible for rollover or purging, so data will grow.

 

Suggestion:- If the customer is not using the "jarvis_jmetrics_1.0" index explicitly, then we can follow these steps:

 

Quick way: 

delete the index  "jarvis_jmetrics_1.0" , so that we can get some disk space. And index will get created automatically. But similar size increase can happen after some months.

 

Full solution:

1)Check if the Template is available {es_url}/_template/jarvis_jmetrics_1.0?pretty

2)If the template is available, need to run the follow steps:

 

a) stop the lean-indexer and indexer.

b) delete the index  "jarvis_jmetrics_1.0"

c) And recreate the index using the below

 

curl command.(if curl having any error, run it using postman):

 

curl --request PUT '{{es_url}}/jarvis_jmetrics_1.0_1' \

 

--header 'Content-Type: application/json' \

--data-raw '{

    "aliases": {

        "jarvis_jmetrics_1.0": {},

        "jarvis_jmetrics_1.0_system": {

            "filter": {

                "term": {

                    "@tenant_id": "system"

                }

            }

        },

        "read_jarvis_jmetrics_1.0": {

            "filter": {

                "bool": {

                    "must": [

                        {

                            "term": {

                                "@doc_type_id": "jmetrics"

                            }

                        },

                        {

                            "term": {

                                "@doc_type_version": "1.0"

                            }

                        }

                    ]

                }

            }

        }

    }

}'

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html