SpectroSERVER shutting down soon after startup trying to allocate XXX bytes of memory

Products

• CA Spectrum

Issue/Introduction

SpectroSERVER shutting down soon after startup trying to allocate XXX bytes of memory

The $SPECROOT/SS?VNM.OUT is showing the following error messages:

Jul 18 13:16:17 ERROR TRACE at VNM.cc(797): S:/CA/SPECTRUM/SS/SpectroSERVER.exe is out of memory while allocating 11 bytes. Scheduling shutdown.

0x113727aa libGlobl.dll!CsSymbolInfo::print_current_stack
0x100aaf59 libsskrnl.dll!SearchManager::process_queued_work
0x11732c5b libPort.dll!Cs_new_handler
0x117364c6 libPort.dll!malloc
0x1136ed82 libGlobl.dll!Cs_strdup
0x7d32f1 libalert.dll!CsSnmpAlert::CsSnmpAlert
0x7d53c2 libalert.dll!TrapQueueNode::TrapQueueNode
0x7d58ea libalert.dll!RemoteForwardingTrapQueue::queue_trap
0x7d628e libalert.dll!CsAlertManagerMT::process_trap_MT
0x7d6676 libalert.dll!CsAlertManagerMT::trap_receiver
0x11ac4cd8 libmoot.dll!Thread::call_thread_function
0x11ac3e76 libmoot.dll!IOEvent::IOEvent
0x7536cbb2 kernel32.dll!CreateFiberEx

Jul 18 13:16:20 ERROR TRACE at VNM.cc(841): C:/SPECTRUM/SS/SpectroSERVER.exe is out of memory while allocating 12 bytes. Continuing shutdown.
CA memory allocation error. Aborting.

Environment

Any Spectrum version

Cause

In this instance, the cause of this excessive memory consumption in a very short time is due to an excessive number of traps that are being sent to the SpectroSERVER.  Spectrum does have trap storm alarming and suppression built in, but Spectrum, as designed, must first read the traps, which uses memory, in order to know that it is part of a trap storm and not a valid trap.  This causes the memory exhaustion and will eventually use up all of the free memory and Spectrum will shutdown as a result.

Resolution

You will need to locate the devices) that are sending an excessive number of traps and resolve the problem that causes them to send the traps to Spectrum.

If it is not known or it is not possible to resolve this problem one option is to temporarily disable all trap processing on the SpectroSERVER.

You can do this by editing the .vnmrc file ($SPECROOT/SS) and changing the option:

from:

snmp_trap_port_enabled

to:

snmp_trap_port_enabled=FALSE

This will allow the SpectroSERVER to start up as it is not processing traps. 

Alternatively you can also change the SNMP Trap Port number in the .vnmrc file

snmp_trap_port=

to

snmp_trap_port=163

This will also stop Spectrum from receiving traps on the default port of 162.

Once the devices have been identified that are sending excessive traps, you can stop the SpectroSERVER and change back these settings to the original values of empty (default).

Additional Information

It is recommended to use Wireshark (Windows) or tcpdump (Linix) on the SpectroSERVER system to examine the trap ports and locate where the traps are coming from.