We use a SANM policy to send alarm information over to our Remedy instance and in some case automatically create tickets. The filter is set to a 2 minute age, and then the SetScript sends some alarm data when they match the filter over to Remedy via a curl. This usually works fine and runs on time. We have noticed that during times of high alarm load it is running many hours behind.
If we kill the process, it kicks back off and starts processing new alarms as they occur, but within a short time it starts falling behind again.
AlarmNotifier configuration
Component : Spectrum Core / SpectroSERVER
By default the AlarmNotifier has a 1 second built in delay for processing alarms. If 60 alarms come in at once, it will take at least 60 seconds for AlarmNotifier to process the alarms. During times of high alarm volume, the Notifier can be delayed due to this. To change this so that the Notifier will process alarms faster, modify the $SPECROOT/Notifier/.alarmrc:
SEND_ALARM_DELAY=1000
Change this to 50 in $SPECROOT/Notifier/.alarmrc so that the Notifier will process an alarm every 50 milliseconds.