Spectrum randomly generates false contact lost alarms and cpu spikes to 100%

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

At random times spectrum will generate thousands of false management agent lost alarms or device has stopped responding to polls alarms.  We are unable to clear these alarms.  If we stop and restart the SpectroSERVER everything goes back to normal.  

We have noticed the SpectroSERVER cpu utilization is near 100%.  When running pstack, we can see in the highest cpu thread "cs_set_ruser_profile_active_status" is at fault:

#157 0x00007fb2455da47c in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#158 0x00007fb2455da47c in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#159 0x00007fb2455da47c in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#160 0x00007fb2455da47c in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#161 0x00007fb2455da47c in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#162 0x00007fb2455da47c in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#163 0x00007fb2455da47c in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#164 0x00007fb2455da50f in AW_TreeForAllNodes () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#165 0x00007fb2455e336b in AW_HashTreeForAllNodes () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#166 0x00007fb245615b49 in cs_set_ruser_profile_active_status () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#167 0x00007fb24561dafd in cs_snmp_incoming_verify () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#168 0x00007fb24561e413 in cs_snmp_incoming_get () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#169 0x00007fb245601d11 in CsHPSERequestSender::DoReadResponse() () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#170 0x00007fb245602c5d in CsHPSEResponseReceiver::DoTask() () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#171 0x00007fb24560ace6 in CsHPSETask::threadRun(void*) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#172 0x00007fb23fa88e25 in start_thread () from /lib64/libpthread.so.0
#173 0x00007fb23f7b634d in clone () from /lib64/libc.so.6

Environment

Release : 10.3

Component : Spectrum Core / SpectroSERVER

Cause

There is a code flaw in the way SNMPv3 profile data is being accessed.

Resolution

This is fixed in Spectrum version 10.4.1.

The following debug patches are solutions to this problem:

There is a patch solution for 10.3.1 in Spectrum_10.03.01.D103 for 10.3.1.  This patch was not built on any bi monthly patch.

There is a patch solution for 10.3.2 in Spectrum_10.03.02.D121 for 10.3.2.  This patch was built on top of bi monthly patch #1 for 10.3.3 - Spectrum_10.03.02.BMP_10.3.201 which means that Spectrum_10.03.02.BMP_10.3.201 must be installed on all SS and OC before installing Spectrum_10.03.02.D121 on the SpectroSERVERs.

Please contact Spectrum support for copies of these patches/media.