Elastic Search - Retention Period

Products

• CA Application Performance Management SaaS

Issue/Introduction

What levels of Retention Period are there for the Elastic Search? 

Environment

Release : SAAS

Component : APM SAAS ENTERPRISE

Cause

Documentation 

Resolution

• Tenant-doc_type level
• Tenant level
• doc_type level
• product level
• purge default

Tenant-doc_type level:
The retention period can be set while onboarding a product tenant or doctype.

Retention Period can be set for a tenant and doctype combination. This takes the highest precedence.
This is done during tenant onboarding.

{
"product_id": "icc",
"tenant_id": "bcci",
"tenant_details": {
"retention_period": 123
},
"doc_type_specific_retention_periods": [
{
"doc_type_id": "D12",
"doc_type_version": "1",
"retention_period": 50
}
]
}

Here purge takes the retention_period as 50

Tenant level
This is done during Tenant Onboarding:

{
"product_id": "icc",
"tenant_details": {},
"tenant_id": "bcci",
"retention_period" : 2,
"ldds": {
"ana": "internal"
}
}

Here purge takes the retention_period as 2

Doc_Type level
This is done during Doc_Type Onboarding:

{
"product_id": "icc",
"doc_type_id": "WC_Schedules",
"doc_type_version": "1",
"unique_id": "$['event_id']",
"retention_period": 50,
"mappings": {
"some_type": {
"properties": {
"event_id": {
"type": "string"
}
}
}
}
}

Here purge takes the retention_period as 50

Product-level
This is done during onboarding a product.

{
"product_description": "Jarvis Product",
"product_details": [
{
"retention_period": 2
}
],
"product_id": "icc",
"product_name": "International cricket council",
"ldds": {
"ana": "internal"
}
}

Here purge takes the retention_period as 2

purge default
If none of the above are set retention period is taken from utils.properties. This can be set as environment variable DEFAULT_RETENTION_PERIOD in ESutils.

Important Note :

• A point to note that one index contains data from many tenants. So unless all the tenant's retention period in a given index is not honored, it will keep the index.