A top command shows that the CPU at 100% utilization and on closer inspection, the SpectroSERVER's SNMP listen thread, is consuming 100% CPU.

Products

• Spectrum
• Network Observability

Issue/Introduction

Seeing the following errors in VNM.out:

WARNING at CsHPSERequestSender.cc(569): SNMP send took 5114ms! Device IP: xxxxx Request ID: xxx Operation Type: 0 Community: xxx

SendRetCode: 0 usReturnCode: 0 Timeout: 3000 Device Port: 161 m_strCommunity: public

m_nVersion: 0 m_strContextName: m_nSecLevel: 0 m_nAuthProtocol: 0

m_strAuthPwd: m_nPrivProtocol: 0 m_strPrivPwd:

WARNING at CsHPSERequestSender.cc(1571): timeout mutex lock took 4835ms!

Environment

DX NetOps Spectrum all currently supported releases

Cause

Excessive number of traps per minute causing a trap storm resulting in processing performance issues on the SpectroSERVER

Resolution

Stop / Kill the SpectroSERVER - NOTE: This will corrupt the SpectroSERVER DataBase (SSdb) so you will need a backup.

Change the snmp trap port = from NULL to 163 in:

$SPECROOT/SS/.vnmrc 

The default trap port on the SpectroSERVER is 162. By setting it to 163 (or any other unused port for that matter), it will no longer detect incoming traps, which will continue to go to the default port.

Initialize the SSdb to legacy

cd $SPECROOT/SS/

../SS-Tools/SSdbload -il legacy.SSdb

Then load a good known SSDB backup 

cd $SPECROOT/SS/

../SS-Tools/SSdbload -il {db_XXXXXX_XX_XXX}.SSdb

Restart Process Daemon

cd $SPECROOT/lib/SDPM
./processd --restart

Start Spectroserver

Now SpectroSERVER will be stable but it will not process the traps.

Find out which devices are sending Trap Storm, either delete those device(s) or remove the trap_destination on the device to this SpectroSERVER

Now change the snmp_trap_port =  (Blank)  in $SPECROOT/SS/.vnmrc

Stop Spectroserver
Restart process daemon
Start Spectroserver