DX O2 Alarms are received late, there is a lag

Products

• DX SaaS
• DX Operational Observability

Issue/Introduction

We have noticed that Alarms are not received on time. There seems to be lag in the system.

Environment

DX O2

Cause

Capacity issue

Resolution

Apply below optimization to reduce LAG 

1. Updated doi-normalized-alarm deployment with below new settings:

-BLOCKING_QUEUE_CAPACITY = 3000 
-ES_DEFAULT_MAX_CONN_TOTAL=50
-THREAD_POOL_SIZE_FOR_ALARM_INGESTION=200

2. Save the changes and wait for doi-normalized-alarm new pod to restart