OI-Notif stops processing traps without warning, logging or alerting
Article ID: 331805
Updated On:
Products
Issue/Introduction
Smarts Service Assurance Manager (Smarts SAM) OI-Notif stops processing traps without warning, logging or alerting
Environment
Cause
From the Smarts SAM OI log file we can see that there is an "out of memory" error reported for the Java heap size for the eciProcessors thread as well as the "EventPoller on notifPublicationQueue" thread:
[April 13, 2013 9:43:08 AM GMT+00:00 +102ms] t@########### platform
MAIN_MSG-*-STDFD_ERR-stderr: Exception in thread "eciProcessors:processor_1" java.lang.OutOfMemoryError: Java heap space
[April 13, 2013 9:47:16 AM GMT+00:00 +735ms] t@########### platform
MAIN_MSG-*-STDFD_ERR-stderr: Exception in thread "EventPoller on notifPublicationQueue" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Resolution
If you encounter this issue, there are a few settings and parameters in Notif that can be tuned to improve the performance of trap processing:
The following sections explain how to adjust each of these Notif settings and parameters.
JVM Memory Size
There are two jvm parameters (-DXmx and -DXms) which define the maximum and minimum committed memory size for the jvm process during startup. The default values are 1024MB and 128MB respectively. These values can be increased based on the system needs.
How to change values:
1. The following asl files needs to be modified with new values.
SAM/smarts/rules/ics/Notif-init.asl
SAM/smarts/rules/icoi/Notif-init.asl2. Add the following lines in asl files, in INIT_JVM rule
jvm->add_prop("-Xms3072m"); //3GB min commit size. Make sure 3GB free RAM space before starting
jvm->add_prop("-Xmx4096m"); //4 GB max value.3. Restart the OI server
Notif ECI Worker Threads size
This setting defines the number of worker threads that will consume events from the NotifEventQueue. The default value of this is 4. As the number of events pile up in the queue, the memory size will increase, increasing the threads will consume those events from queue and can prevent memory growth.
How to change values:
1. Edit the following import file (local/conf/icoi/Notif_Settings.import)as shown below:
Notif_Settings::Notif-Settings
{
Active=TRUE
PendingUpdate=FALSE
LogSize=8388608
NbrLogFiles=5
LogLevel=2
MaxEnumWildcard=4
ProcessCount=4 // Defines the thread size; Increase this value to suitable value based on the number of processors.
}2. Restart the OI server
NotifEventQueue Size limit
This setting defines the max events that can be present in the queue. The events exceeding this limit will be discarded. Currently there is neither logging nor any notifications being generated when events are discarded due to the QueueSize limit. By default, this queue is unbounded and set to maximum.
How to change the value:
1. Add the following line to the Notif_Settings.import file (local/conf/icoi/Notif_Settings.import) as shown below:
call notifInterface::notifEventQueue setMaxSize 500002. Restart the OI server.
Feedback
