A top command shows that the CPU at 100% utilization and on closer inspection, the SpectroSERVER's SNMP listen thread, is consuming 100% CPU.

Products

• CA Spectrum
• CA eHealth

Issue/Introduction

Seeing the following errors in VNM.out:

WARNING at CsHPSERequestSender.cc(569): SNMP send took 5114ms! Device IP: xxxxx Request ID: xxx Operation Type: 0 Community: xxx

SendRetCode: 0 usReturnCode: 0 Timeout: 3000 Device Port: 161 m_strCommunity: public

m_nVersion: 0 m_strContextName: m_nSecLevel: 0 m_nAuthProtocol: 0

m_strAuthPwd: m_nPrivProtocol: 0 m_strPrivPwd:

WARNING at CsHPSERequestSender.cc(1571): timeout mutex lock took 4835ms!

Environment

Release : 10.3
Component : Spectrum Core / SpectroSERVER

Resolution

Stop / Kill the SpectroSERVER

Change the snmp trap port = from NULL to 163 in:

$SPECROOT/SS/.vnmrc 

Initialize to legacy

cd $SPECROOT/SS/

../SS-Tools/SSdbload -il legacy.SSdb

Then load a good know SSDB backup

cd $SPECROOT/SS/

../SS-Tools/SSdbload -il {db_XXXXXX_XX_XXX}.SSdb

Restart Process Daemon

cd $SPECROOT/lib/SDPM
./processd --restart

Start Spectroserver

Now spectroserver will be stable but it will not process the traps.

Find out which devices are sending Trap Storm, either delete those device(s) or remove the trap_destination on the device to this spectroserver

Now change the snmp_trap_port=  (Blank)  in $SPECROOT/SS/.vnmrc

Stop Spectroserver
Restart process daemon
Start Spectroserver