Cleared Cisco APIC alarms return to Spectrum daily from VNA

Products

• DX NetOps

Issue/Introduction

Cleared Cisco APIC alarms return to Spectrum after clearing them. They do not exist anymore in Cisco APIC.

Cause

It is possible that alarms were cleared from APIC during VNA downtime (e.g. during NetOps upgrade) .  If the ACI engine is down and any alarms are cleared during down time APIC will not send clear alarms again as the alarms will be deleted from APIC itself.  The alarms that are cleared during VNA downtime need to be manually delete from VNA MySQL DB.

Resolution

We added a functionality in 22.2.9 that should prevent or work around this problem, where If there is VNA down time of greater than 60 minutes, the aci plugin would perform a full sync. 

"SYNC_ALARMS_AFTER_DOWNTIME_IN_MINUTES": 60
 
The default is 60 mins but this can be configured as follows:
https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/virtual-network-assurance/23-3/building/cisco-aci.html#concept.dita_cba8403d483bfb339ef1ec63bfff4124db20fd8c_pluginconfig

 
For downtimes of less than 60 mins (or whatever you configure) or on versions prior to 22.2.9, Engineering has recommended the following.
 
1. stop the engine
2. delete Alarms_LastSyncTime file from /opt/CA/VNA/collector/ACI_UUID/repository/work/subscription
3. start the engine
 
this will force a sync of the alarms and clears from APIC.