SpectroSERVER process consumes 100% cpu processing traps

Products

• Network Observability
• Spectrum

Issue/Introduction

The SpectroSERVER stops responding properly and you may see the SS disconnect in OneClick.  When reviewing "top" output we can see that the SpectroSERVER is at 100% cpu utilization.  As root user, run "pstack" on the pid of the SS and review Thread #1.

#4  0x00007f90e58b5ee8 in CsAlertFwdMgr::forward_trap_to_remote_lscpe(CsModelHandle const&, CsDcmIfBlk const&, int) () from /opt/SPECTRUM/lib/../SS/libalert.so.1
#5  0x00007f90e58b974a in CsAlertFwdMgr::forward_trap_to_remote_servers(CsDcmIfBlk const*, CsSnmpAlert const*, bool*, bool*, bool*) () from /opt/SPECTRUM/lib/../SS/libalert.so.1
#6  0x00007f90e58c0ca8 in CsAlertManagerMT::process_trap_to_forward(TrapQueueNode const*) () from /opt/SPECTRUM/lib/../SS/libalert.so.1

Cause

Too many traps being sent to the SpectroSERVER.  The SpectroSERVER needs to process them and in this case Spectrum's Trap Director was enabled which "costs" Spectrum more resources as it needs to determine which SS to send the information to.

Resolution

Reduce the number of traps being sent to the SpectroSERVER.  On average the SS can handle 300 traps/sec sustained (This number can vary depending on many factors.  If you are unsure please open a support ticket).