Elasticsearch node space issue

Products

• DX Operational Intelligence

Issue/Introduction

After enabling AXA and DX OI integrations with UIM, the Elasticsearch environment is growing rapidly. We have already done some workarounds but the environment continues to consume the entire disk, which is currently at above 2 TB. We would like some help to verify if we can make this growth slow or what would be the most suitable disk environment for the environment.

Environment

Release : 22.1

Cause

We find that the index ao_apm_tt_2 size is around 2TB. The index was deleted last Friday. It seems like in one week the index got that big. It seems like move data may not be better option as the size of the index is that big. Also, it looks like index ao_apm_tt_2 is not onboarded properly.

Resolution

We onboarded the index ao_apm_tt_2 again by taking the following steps.

 

Scale down the 3 deployments of apmservices-tracestore...

Confirmed that no additional documents being added in the index ao_apm_tt_2.

Deleted the index ao_apm_tt_2.

Created/onboarded new index using the information from <DX-install>/post-install/3.apm.onboard.sh,

We made one modification. Replaced:

Replaced: "unique_id": "$['"'"'summary'"'"']['"'"'traceId'"'"']",

With: "unique_id": "$['summary']['traceId']", 

This time the index got created/onboarded properly. 

Scale up the 3 deployments of apmservices-tracestore...

Scale down the deployments of jarvis util and jarvia kron

Scale up the deployments of jarvis util and jarvia kron