SpectroSERVER CPU at constant 100%
Article ID: 182960
Updated On:
Products
Issue/Introduction
The SpectroSERVER process is constantly using 100% CPU (100% of 1 processor). This causes slowness when navigating the landscape resulting in poor responsiveness in OneClick.
Environment
Release : 10.3.x, 10.4.x, 20.x
Component : SpectroSERVER
Cause
The SpectroSERVER was being flooded with SNMPv3 traps from one specific device. The volume of traps was such that the SpectroSERVER was continually processing trying to keep up with the volume.
Resolution
Reconfigure the identified device to only send traps that are needed and stop the continual flood that was occurring. Once done, the SpectroSERVER CPU utilization returns to normal.
Additional Information
If devices are also being polled via SNMPv3 you may see continual messages like this in the VNM.OUT
Oct 01 09:45:13 WARNING at CsHPSERequestSender.cc(569): SNMP send took 1024ms! Device IP: xx.xxx.xxx.xxx Request ID: 21227 Operation Type: 0 Community: #v3/P:SHA^authpassword#wq:AES^privpass/snmpUser
SendRetCode: 0 usReturnCode: 0 Timeout: 60000 Device Port: 161 m_strCommunity: WDCS
m_nVersion: 3 m_strContextName: m_nSecLevel: 3 m_nAuthProtocol: 2
m_strAuthPwd: authpassword m_nPrivProtocol: 2 m_strPrivPwd: privpassword
Oct 01 09:45:15 WARNING at CsHPSERequestSender.cc(569): SNMP send took 1566ms! Device IP: xx.xxx.xxx.xxx Request ID: 21228 Operation Type: 0 Community: #v3/P:SHA^authpassword#wq:AES^privpass/snmpUser
SendRetCode: 0 usReturnCode: 0 Timeout: 60000 Device Port: 161 m_strCommunity: WDCS
m_nVersion: 3 m_strContextName: m_nSecLevel: 3 m_nAuthProtocol: 2
m_strAuthPwd: authpassword m_nPrivProtocol: 2 m_strPrivPwd: privpassword
Oct 01 09:45:16 WARNING at CsHPSERequestSender.cc(569): SNMP send took 1018ms! Device IP: xx.xxx.xxx.xxx Request ID: 20995 Operation Type: 0 Community: #v3/P:SHA^authpassword#wq:AES^privpass/snmpUser
SendRetCode: 0 usReturnCode: 0 Timeout: 60000 Device Port: 161 m_strCommunity: wdcs
m_nVersion: 3 m_strContextName: m_nSecLevel: 3 m_nAuthProtocol: 2
m_strAuthPwd: authpassword m_nPrivProtocol: 1 m_strPrivPwd: privpassword
Oct 01 09:45:16 WARNING at SnmpItcInterface.cc(156): SNMP failsafe - Retrying request( IP=xxx.xxx.xxx.xxx, community=theOffice, id=20863) after 360000ms.
Initially using the SysInternals Procdump to generate a series of dmp files to analyze shows the SpectroSERVERs moot thread to be sitting idle.
ntdll!NtWaitForSingleObject+0xa
KERNELBASE!WaitForSingleObjectEx
libmoot!CsMoot::suspend
libmoot!CsMootBase
libmoot!spin
libmoot!call_func
kernel32!CreateFiberEx
Using SysInternals Process Explorer utility, the problem thread is found:
Right Click ~~> Properties ~~> Threads. (TID 3996 is using 48%+ constantly)
This allows the stack for the thread to obtained and then it can be seen that it is constantly processing traps in the TrapReceiver:
ntoskrnl.exe!_misaligned_access+0x13fd
ntoskrnl.exe!KeSynchronizeExecution+0x4552
libcaopenssl_crypto.dll!caopenssl_cryptolib_get_interface+0x75b19
libcaopenssl_crypto.dll!caopenssl_cryptolib_get_interface+0x765dd
libcapki.dll!capki_lib_get_interface+0x7d278
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x3b97
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x1c714
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x1c90f
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x28c97
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x27570
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x2ceda
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x2bbac
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x21807
libhpse.dll!TrapxTrapReceiver::initialize_socket+0x1dc41
libhpse.dll!CsHPSETrapReceiver::DoTask+0xb64
libhpse.dll!CsHPSETrapReceiver::DoTask+0xf5
libhpse.dll!CsHPSETask::threadRun+0x3d
KERNEL32.DLL!BaseThreadInitThunk+0x22
ntdll.dll!RtlUserThreadStart+0x34
Using WireShark to packet capture, the trace (capture filter 'port 162') shows that what specific device/s is flooding the SpectroSERVER with non-stop SNMP v3 traps.
Once this device/s is reconfigured to not send traps, the SpectroSERVER's CPU utilization returns to normal
Feedback
