SANM Alarm Notifier getting hours behind during times of high system load

Products

• Spectrum
• Network Observability

Issue/Introduction

We use a SANM policy to send alarm information over to our Remedy instance and in some case automatically create tickets. The filter is set to a 2 minute age, and then the SetScript sends some alarm data when they match the filter over to Remedy via a curl.  This usually works fine and runs on time.  We have noticed that during times of high alarm load it is running many hours behind.  

If we kill the process, it kicks back off and starts processing new alarms as they occur, but within a short time it starts falling behind again.

Environment

All supported DX NetOps Spectrum releases

Cause

AlarmNotifier configuration limiting ability to keep up with alarm processing when times of increased alarm activity are present.

Resolution

By default AlarmNotifier has a 1 second built in delay for processing alarms.

If 60 alarms come in at once, it will take at least 60 seconds for AlarmNotifier to process the alarms.

During times of high alarm volume the Notifier can be delayed in alarm processing due to this configuration.

To change this allowing Notifier to process alarms faster modify the $SPECROOT/Notifier/.alarmrc file. Find the default entry set to 1000ms (one second).

SEND_ALARM_DELAY=1000

Change this to a value of 50 allowing Notifier to process an alarm every 50 milliseconds.

Edit the .alarmrc file and save the change. Restart AlarmNotifier to have it read in the new configuration. The changes go into effect only after AlarmNotifier is restarted.