Spectrum AlarmNotifier sending alarms with hours of delay

Products

• Spectrum

Issue/Introduction

We have on the MLS 2 instances of AlarmNotifier sending alarms to different platforms. With one platform, the alarms get there almost immediately, but in the other, alarms are 2 or 3 hours late.

Environment

Release : 10.4

Component : Spectrum Core / SpectroSERVER

Cause

looking in notifier.out we see connection problems with the various archive managers.
This is needed for notifier to get event history, before it can forward alarms so may be causing the delay.

Line 80: AlarmNotifier: Warning: 08:47:58 PM 06/03/2020 - Global Alarms on xxx: connection to SpectroSERVER lost.
Line 333: AlarmNotifier: Warning: 08:50:03 PM 06/03/2020 - Events on yyy: the Archive Manager has been shut down.
Line 628: AlarmNotifier: Warning: 08:52:18 PM 06/03/2020 - Global Alarms on xxx: connection re-established.
Line 809: AlarmNotifier: Warning: 08:53:38 PM 06/03/2020 - Events on yyyy: connection re-established.
Line 10586: AlarmNotifier: Warning: 09:23:14 PM 06/03/2020 - Events on zzz SPC-SHD-23008: : the Archive Manager is not responding.
Line 10741: AlarmNotifier: Warning: 09:23:54 PM 06/03/2020 - Events on zzz: connection re-established.
Line 10860: AlarmNotifier: Warning: 09:24:28 PM 06/03/2020 - Events on xxxx SPC-SHD-23008: : the Archive Manager is not responding.
Line 11003: AlarmNotifier: Warning: 09:25:08 PM 06/03/2020 - Events on xxx: connection re-established.
Line 37916: AlarmNotifier: Warning: 11:27:48 PM 06/03/2020 - Events on zzz SPC-SHD-23008: : the Archive Manager is not responding.

Resolution

This points to networking issues between notifier and the various archive managers. 

workaround:

archive manager dependence can be disabled to test if it is the problem.
Please be aware that you will lose event history in your alarms.

SHOW_ONLY_ORIGINATING_EVENT - This parameter essentially disables the AlarmNotifier's use of the Archive Manager, and therefore can only display the event that originated the alarm. If this option is set, the Store_Originating_Event attribute( 0x1296f ) must be set to TRUE( the default ) on the AlarmMgmt model of every SpectroSERVER. False( the default value ) or True.