Spectrum AlarmNotifier is running but appears to be hung

Products

• Spectrum
• Network Observability

Issue/Introduction

Spectrum AlarmNotifier is running but appears to be hung. 

Environment

Release : Any release

Component : SPCAPP - Spectrum Applications

Cause

Running a process list on Linux shows the following:

spectrum   14319   3881  0 Apr25 ?        00:00:28 /app/SPECTRUM/Notifier/AlarmNotifier

spectrum 102439  14319  0 May06 ?        00:00:00 sh -c /app/SPECTRUM/Notifier/ClearScript 05/06/2020 12:57:25 "GnSNMPDev" "Router1" "30131825" "MAJOR" "3b70031" "" "spectrum01" "0x500000" "0x5accb3" "0x3d0002" "xx.xx.xxx.xxx" "" "NEW" "FALSE" "TRUE" "FALSE" "" "0" "" "14319" "Y" "F5DV"

From the above output, we see the AlarmNotifier process is running. However, there is also a ClearScript process running that showed to be running for over 12 hours.

The ClearScript is hung causing the AlarmNotifier process to hang as well.

Resolution

Killing the process id of the hung ClearScript allowed the AlarmNotifier process to continue processing alarms.

Additional Information

Try stopping the AlarmNotifier via Install Ticket:

How can I start and stop processes that are running under processd in Spectrum?

cd $SPECROOT/bin/

./cmdC localhost 2 ALARMNOTIFIER

If the AlarmNotifier does not stop, check the SetScript, UpdateScript, and ClearScript. The script may be calling a third party utility that is hung.

For example: In the SetScript, it is calling a third party utility snsClient.exe to send SMS.

Kill the third party utility that is hung.

Rerun: ./cmdC localhost 2 ALARMNOTIFIER

Once the AlarmNotifier is successfully stopped, re-launch it.

cd $SPECROOT/bin/

./launchinstdbapp localhost ALARMNOTIFIER y NOTIFIER.OUT