Spectrum Server crashes or cpu hangs at 100% in v3 code

Products

• CA Spectrum
• CA eHealth

Issue/Introduction

The SpectroSERVER hangs at 100% cpu and eventually crashes.  Stack dumps ($SPECROOT/SS/support) and/or core file stacks show that Spectrum is processing v3 data:

#1274 0x00007f8ccf5c59f6 in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1275 0x00007f8ccf5c59f6 in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1276 0x00007f8ccf5c59f6 in _AW_TreeForAllNodesRecursive () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1277 0x00007f8ccf5c5ab6 in AW_TreeForAllNodes () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1278 0x00007f8ccf5cdbab in AW_HashTreeForAllNodes () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1279 0x00007f8ccf5f5459 in cs_search_remote_user () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1280 0x00007f8ccf5f8e92 in cs_find_user_by_name_exact () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1281 0x00007f8ccf5fda08 in cs_snmpv3_process_incoming.isra.0 () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1282 0x00007f8ccf5fe09b in cs_snmp_parse_v3_pdu () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1283 0x00007f8ccf5ee925 in cs_snmp_parse_pdu () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1284 0x00007f8ccf5f2dbc in cs_snmp_incoming_get () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1285 0x00007f8ccf5a8067 in CsHPSERequestSender::DoReadResponse() () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1286 0x00007f8ccf5a9a3d in CsHPSEResponseReceiver::DoTask() () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

#1287 0x00007f8ccf5b28b6 in CsHPSETask::threadRun(void*) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1

Environment

Component : Spectrum Core / SpectroSERVER

Cause

Misconfigured autodiscovery configuration

Resolution

The cause was an autodiscovery scan that was set to run every morning and discover 0.0.0.0 through 255.255.255.255 - even though it wouldn't model anything, Spectrum would take the addresses that responded to ping and add an entry to the v3 cache. After the scan was done, there would be 2 million entries in the cache, which caused a performance issue. 


Best Practice and proper discovery configuration is to use a limited range discovery or a seed router so that Spectrum doesn't scan everything.


https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/spectrum/10-4-1/managing-network/modeling-and-managing-your-it-infrastructure/discovering-and-modeling-your-network/define-a-discovery-configuration.html#concept.dita_5639928139af469861f7378917d8184a04747c76_SeedRouters

Additional Information

Please take note that this is different than the v3 issue documented here:


https://knowledge.broadcom.com/external/article?articleId=138088