Alarms are delayed and alarm_enrichment queue is backed up with messages.

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

The alarm_enrichment queue is processing but not keeping up and getting more and more messages in the queue. Alarms showing up in the console are behind with new alarms showing up that were generated hours ago.

 

Environment

Release: UIM 20.x

Component: Alarm Enrichment

Cause

There are multiple factors that could be resulting in this problem.

Check C:\Program Files (x86)\Nimsoft\probes\service\nas\alarm_enrichment\alarm_enrichment.log 

Potentially there could be a problem alarm, an out of memory condition, or something else.

Samples:

Jul 13 15:35:55:299 [pool-1-thread-20, alarm_enrichment] InboundAlarmProcessor:Error: processing alarm: cc3... 
Jul 13 15:35:55:299 [pool-1-thread-20, alarm_enrichment] java.lang.OutOfMemoryError: GC overhead limit exceeded 

No element found for key level

Resolution

For error "No element found for key level":

In the case of a problem alarm the Dr.NimBUS utility can be used to run the messages in the queue, locate and delete the problem alarm message from the queue.

Use the Dr. Nimbus tool to fetch unread messages from the alarm_enrichment queue. If the messages are missing any fields other than md5sum, or if they do not match the expected Subject, then those messages could be holding the queue up. To access the DrNimBUS tool, on the primary hub select Start button->All Programs->Nimsoft Monitoring and then Tools->DrNimBUS.

Use this button to use the tool on the primary hub:

Choose a queue, and then fetch single messages using the + button.

Considerations:

1) It helps to set the bulk size on nas/ alarm_enrichment down to 10. If the probe still cannot load the first ten messages, then you know one or more of the first ten messages is causing an issue.

2) If the messages contain a subject of QOS_MESSAGE, then they are the incorrect subject and cannot be processed by alarm_enrichment. Fetch all QOS_MESSAGEs out until the subjects are alarm or filter on alarm as a Subject.

3) The DrNimbus tool permanently marks the messages as read. Once you see them in DrNimbus, they cannot be delivered to the probe.

4) Using Dr. Nimbus to read the alarm_enrichment queue may disconnect alarm_enrichment. In this case, the queue will show as yellow, and the probe will need to be restarted.

For Memory issues:

Memory allocation for alarm_enrichment can be modified via selecting the probe from the IM Console > right click Edit > Arguments

The default is:

-Xms64m -Xmx1024m -Dfile.encoding=UTF-8 -jar ../lib/alarm_enrichment.jar 

Check system memory usage to ensure there is enough free RAM (4gb free after the increase), such as for Windows via Task Manager.

Provided there is enough free memory the probe allocation can be increased.

-Xms1024m -Xmx2048m -Dfile.encoding=UTF-8 -jar ../lib/alarm_enrichment.jar 
or 
-Xms2048m -Xmx4096m -Dfile.encoding=UTF-8 -jar ../lib/alarm_enrichment.jar 

After making the change deactivate and then activate the probe.

Additional Information

Normally, the optimal alarm_enrichment queue bulk_size value is 100.

To set it open the hub probe raw configure (in IM Console via Shift + right click)  
Expand the tree under 'postroute' > select alarm_enrichment > select bulk_size > click Edit Key > Enter New Value: 100 > click Apply.  
Note this will result in a restart of the hub so this may need to be done after hours.