SANM Alarm Notifier getting hours behind during times of high system load

Products

• CA Spectrum

Issue/Introduction

We use a SANM policy to send alarm information over to our Remedy instance and in some case automatically create tickets. The filter is set to a 2 minute age, and then the SetScript sends some alarm data when they match the filter over to Remedy via a curl.  This usually works fine and runs on time.  We have noticed that during times of high alarm load it is running many hours behind.  

If we kill the process, it kicks back off and starts processing new alarms as they occur, but within a short time it starts falling behind again.

Cause

AlarmNotifier configuration

Environment

Component : Spectrum Core / SpectroSERVER

Resolution

By default the AlarmNotifier has a 1 second built in delay for processing alarms.  If 60 alarms come in at once, it will take at least 60 seconds for AlarmNotifier to process the alarms.  During times of high alarm volume, the Notifier can be delayed due to this.  To change this so that the Notifier will process alarms faster, modify the $SPECROOT/Notifier/.alarmrc:

SEND_ALARM_DELAY=1000

Change this to 50 in $SPECROOT/Notifier/.alarmrc so that the Notifier will process an alarm every 50 milliseconds.