DX OI - Many alarms are cleared in Spectrum however they remain opened in Alarm Analytics

book

Article ID: 224809

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

We are observing that many alarms are not cleared on DX OI.  Is there any way to synchronize DX OI and Spectrum?

Cause

1. Alarm Reconciliation disabled.

2. doi-normalized-alarm defect in 20.2.x: OI is missing to process updates from alarms_spectrum index, this can happen if OI has to process newly and closed alarms within a few seconds in Spectrum. SpectrumDataPublisher can pull a maximum of 10K alarms from OI which is an OI limitation. So in case if the customer has more than 10K alarms in OI [including stale], SpectrumDataPublisher may not be able to clear all the stale alarms in one restart as it is not guaranteed to receive all the stale alarms in the 10K alarms pulled. In this case, multiple restarts are required until all the stale alarms are cleared in OI

Environment

DX Operational Intelligence 20.2.x 

Spectrum Data Publisher 10.4.2.2

Resolution

Recommendations:

1. Make sure Alarm Reconciliation in enabled in Spectrum Publisher config.xml

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/10-4/integrating/ca-spectrum-and-analytics/integrate-ca-spectrum-with-ca-digital-operational-intelligence-using-spectrum-data-publisher.html#concept.dita_01d23b21ebdeb3555223a11e764ef8c41f86b5f7_UpgradeSpectrumDataPublisher

2. Workaround known defect:  Change AlarmReconcileInfo in the DOIConfig.xml to pull alarms from alarms_all index. This runs reconciliation during SpectrumDataPublisher startup by pulling alarms from DOI[alarms_all] and Spectrum, then deleting the stale alarms in OI. 

a) Update SpectrumDataPublisher/config/DOIConfig.xml
   from
 
<AlarmReconcileInfo> 
    <AlarmsSearchPath>mdo/v2/aoanalytics/alarms/alarms_spectrum/_search</AlarmsSearchPath>
    <QueryParam>@doc_type_id:itoa_alarms_spectrum</QueryParam>
    <QueryParam>status:(NOT CLOSED)</QueryParam>
    <QueryParam>product:Spectrum</QueryParam>
    <QueryParam>@tenant_id:</QueryParam> <!-- This value will be set with DestinationDefaultTenant value -->
</AlarmReconcileInfo>
 
to
 
<AlarmReconcileInfo> 
    <AlarmsSearchPath>mdo/v2/aoanalytics/alarms/alarms_all/_search</AlarmsSearchPath>
    <QueryParam>doc_type_id:itoa_alarms_spectrum</QueryParam>
    <QueryParam>status:(NOT CLOSED)</QueryParam>
    <QueryParam>product:Spectrum</QueryParam>
    <QueryParam>@tenant_id:</QueryParam> <!-- This value will be set with DestinationDefaultTenant value -->
</AlarmReconcileInfo>
 
b) Get the DX Auth Token Used in Alarm Reconcile Configuration
 
 
c) Update SpectrumDataPublisher/config/ConnectorConfig.xml with new DX Auth Token
 
d) Start connector (run.bat/sh start)

** IMPORTANT **: SpectrumDataPublisher can pull a maximum of 10K alarms from OI which is an OI limitation. So in case if the customer has more than 10K alarms in OI [including stale], SpectrumDataPublisher may not be able to clear all the stale alarms in one restart as it is not guaranteed to receive all the stale alarms in the 10K alarms pulled. In this case, multiple restarts are required until all the stale alarms are cleared in OI

Additional Information

https://knowledge.broadcom.com/external/article/190815/dx-aiops-troubleshooting-common-issues-a.html