Ionix/Smarts SAM Adapters: SNMP Trap Adapter keeps crashing from running out of memory

Products

• VMware

Issue/Introduction

Symptoms:




Ionix SAM/Smarts SAM SNMP Trap Adapter keeps crashing from running out of memory.

Error message in Ionix SAM/Smarts SAM SNMP Trap Adapter log file:

[26-Aug-2010 09:46:11 AM+845ms EDT] t@5 SNMP_TrapsHandler [0.0.0.0:162] Receiver
CI-W-EWHILE-While executing function "receivePacket"; in file
   "/work/whitecurrent/FOUNDATION-7.2.0.X/7/smarts/snmp/lib/SNMP_Receiver.c"
   at line 118
SNMP-ENOMEM-Out of memory

Environment

VMware Smart Assurance - SMARTS

Cause

A trap flood is causing the SAM SNMP Trap Adapter process to run out of memory.

Resolution

The following sections provide troubleshooting actions and commands recommended to help you find out and resolve why the trap adapter process is running out of memory.

Confirming trap flood issue
To check for the trap flood issue, do the following:

1. Run the following dmctl command:
   
   <BASEDIR>/SAM/smarts/bin/dmctl -s <TRAP Adapter> get SNMP_TrapManager::SNMP-Trap-Manager
    
2. Look for TrapRate value. This value represent the number of traps processed per minute and should be less than 600. A value greater than 600 indicates a trap flooding issue.

Ionix SAM 8.x/9.x: Determining the source of trap flood
To determine the source of trap flood on Ionix SAM 8.x/9.x environments, turn on logging in the environment as follows:

1. Open the trap_mgr.conf configuration file for editing using sm_edit:
   
   <BASEDIR>/SAM/smarts/bin/sm_edit conf/icoi/trap_mgr.conf
    
2. Find the following entry: 
   
   #DEBUG = FALSE
   #LOGGING = OFF
    
3. Uncomment the above line (remove the "#") and change the above entry to the following:
   
   DEBUG = TRUE
   LOGGING = ALL
    
4. To apply this change in the active environment, do either of the following:
5. • Restart the trap adapter OR
   • Run the following reload command:
     
     <BASEDIR>/SAM/smarts/bin/sm_trapd -s <TRAP Adapter> --reloadConfig

IMPORTANT! Remember to set LOGGING to OFF after issue is resolved.

Ionix SAM 7.x: Determining the source of trap flood
To determine the source of trap flood on Ionix SAM 7.x environments (formerly Smarts SAM), turn on debug in the environment as follows:

1. Open the trap_mgr_parse.asl configuration file for editing using sm_edit:
   
   <BASEDIR>/SAM/smarts/bin/sm_edit rules/icoi-trapd/trap_mgr_parse.asl
    
2. Find the following entry: 
   
   debug = FALSE;
    
3. Change the above entry to the following:
   
   debug = TRUE;
    
4. To apply this change in the active environment, you must restart the trap adapter.
   
   IMPORTANT! Remember to set debug back to FALSE after issue is resolved.

Identifying trap IP addresses in SAM environments
In SAM environments, you can identify IP addresses that the trap adapter is processing by running grep commands (Unix) or findstr commands (Windows).

Unix "grep" command example:
grep Agent TRAP-INCHARGE-OI.log | grep -v CREATE | awk '{print $2}' | sort | uniq -c | sort -n

The above grep command will show how many traps the trap adapter is processing. There are many different grep commands depending on the trap adapter log file, so use a command that best suits your environment.