Is it possible to filter the data to store?(default 45 days)
Requirement is to filter some fields to optimise the storage - on Elastic there are a lot of columns that are stored but it would be possible to remove some of the data fields to optimize storage.
Release : 21.3
Component : CA DOI AO PLATFORM COMPONENTS
This is not possible, data retention is only possible down to index level
This is some information on data retention settings at various levels:
Tenant_doc-type means the settings for a specific combination tenant and doc_type. That setting is not valid for other tenants in that doc_type (index) or for that tenant and any other doc_type.
The order of consideration of retention period by Jarvis while performing operations, in decreasing order, is as follows:
tenant+doctype > tenant > doctype > product > default (taken from utils.properties)
This is an example for quick reference.
Consider 3 Tenants 1,2,3, indices ao_itoa_alarms_all_1_1 and ao_itoa_alarms_apm_1_1 and the product ao for this example and assume the following steps have been performed. Please note that tenants 1,2,3 refer to Cohort ID and not the tenant name.
Now, this will be the final table for product ao
|tenant (cohort-id||itoa_alarms_all_1_1||itoa_alarms_apm_1_1 (and other indicdes)|
As for defining the retention period after onboarding, it can be done at the tenant and docType (mapping in Jarvis swagger UI) level using the PATCH method. tenant and tenant+doctype combinations are set using the tenant onboarding endpoint whereas doctype level retention is set using doctype onboarding endpoint. This can also be done using a rest client like Postman. A sample payload for tenant+doctype retention period from our example is given below.