Delete parts in Elastic to maximize space

Products

• DX Operational Intelligence

Issue/Introduction

Is it possible to filter the data to store?(default 45 days)

Requirement is to filter some fields to optimise the storage - on Elastic there are a lot of columns that are stored but it would be possible to remove some of the data fields to optimize storage.

Environment

Release : 21.3

Component : CA DOI AO PLATFORM COMPONENTS

Cause

This is not possible, data retention is only possible down to index level

Resolution

This is some information on data retention settings at various levels:

 

Tenant_doc-type means the settings for a specific combination tenant and doc_type. That setting is not valid for other tenants in that doc_type (index) or for that tenant and any other doc_type.

The order of consideration of retention period by Jarvis while performing operations, in decreasing order, is as follows:

tenant+doctype > tenant > doctype > product > default (taken from utils.properties)

 

This is an example for quick reference.

 Consider 3 Tenants 1,2,3, indices ao_itoa_alarms_all_1_1 and ao_itoa_alarms_apm_1_1 and the product ao for this example and assume the following steps have been performed. Please note that tenants 1,2,3 refer to Cohort ID and not the tenant name.

 

• The default OOTB retention period, for all tenant, docTypes and products, is 45 days, as mentioned earlier. This is the initial state.
• Create product ao with a retention period of 30 date. This is now applicable for all tenants and indices for product ao.
• Onboard a docType ao_itoa_alarms_all_1_1, specify a retention period of 15 days. Now, this doctype has 15 days retention period while other indices in ao product have 30 days.
• Create a tenant 1 and set retention to 10 days during tenant creation. This sets retention period for this tenant on all doctypes and supercedes the retention period of 15 days which was created during docType onboarding.
• Create tenant 2 without setting a retention period for product ao. For this tenant all docTypes will have retention period of 30 days except ao_itoa_alarms_all_1_1 which will have 15 days (since doctype retention period is there but tenant specific retention period is not defined).
• Create tenant 3, and set retention period on ao_itoa_alarms_all_1_1 index for tenant 3 to 7 days and general retention period to 10. Now for this tenant, only ao_itoa_alarms_all_1_1 index has 7 days retention period and other indices for this tenant have 30 days retention period (Since retention period for this tenant is defined only for a specific index).

 

Now, this will be the final table for product ao

tenant (cohort-id	itoa_alarms_all_1_1	itoa_alarms_apm_1_1 (and other indicdes)
1	10	10
2	15	30
2	7	10

 

 

 

 

 

As for defining the retention period after onboarding, it can be done at the tenant and docType (mapping in Jarvis swagger UI) level using the PATCH method. tenant and tenant+doctype combinations are set using the tenant onboarding endpoint whereas doctype level retention is set using doctype onboarding endpoint. This can also be done using a rest client like Postman. A sample payload for tenant+doctype retention period from our example is given below.

{

    "product_id": "ao",

    "tenant_details": {},

    "tenant_id": "tenant",

    "retention_period": 10,

    "tenant_doc_type_details": [

        {

            "doc_type_id": “itoa_alarms_all”,

            "doc_type_version": "1",

            "retention_period": 7

        }

    ]

}