OI SaaS Spectrum alarms not cleared

Products

• DX Operational Intelligence

Issue/Introduction

 

 

OI SaaS reports alerts but those alerts were already closed in Spectrum

 

 

Environment

Release : 1.3.2

Component : CA DOI SPECTRUM DATA PUBLISHER

Cause

 

 

Observed that all recent alarms are getting closed in alarms_all index. However some old spectrum alarms are in CLOSED state in alarms_spectrum index, but they are still in OPEN status. 

Due to threading issue alarm update ( service enrichment ) / alarm ingestion (as partial update from source product ) which comes at the same time, this can cause discarding one of the updates.

This was observed in other issues as well.

Resolution

 

 

 

To handle this problem, locking ( at cache in normalized alarm service )  has been handled, so that only one update will be handled at a time from cache entry.  

Has been deployed to OI SaaS environment