SpectroSERVER hangs when loading a production db into a lab server

Products

• Network Observability
• Spectrum

Issue/Introduction

When loading a SpectroSERVER db from production into a lab, and that lab box does not have SNMP access to the production device network, the SS will hang and the SS cpu will be stuck at 100%.

pstack of the SS process (as root, ./pstack pidofSS) in thread #1 show alarm impact:

AlarmImpactService 

CsIHFIOverSeer::client_domain_alarm_rem_notify(FINotificationClient*, CsGlobalAlarmID*, CsModelHandle const&) [/opt/SPECTRUM/lib/../SS/libmdlsvint.so.1]
CsFaultRecord::rem_domain_alarm(CsModelHandle const&, bool) [/opt/SPECTRUM/lib/../SS/libmdlsvutil.so.1]
CsIHFIOverSeer::record_merge(CsFaultRecord*, CsFaultRecord*, CsModelHandle const&) [/opt/SPECTRUM/lib/../SS/libmdlsvint.so.1]
CsIHFIOverSeer::add_to_table(CsModelHandle const&, CsModelHandle const&, CsFaultRecord*) [/opt/SPECTRUM/lib/../SS/libmdlsvint.so.1]
CsIHFIOverSeer::add_entries(CsModelHandle const&, CsModelHandle const&, CsULongList*) [/opt/SPECTRUM/lib/../SS/libmdlsvint.so.1]

Cause

No SNMP access to the production network causes Spectrum to try and calculate the alarm impact for thousands of devices which overloads the SpectroSERVER.

Resolution

Edit the $SPECROOT/custom/Events/EventDisp and add this entry to prevent "Device Has Stopped Responding to Polls" alarms:

0x10d35 E 75

This should only be done in a lab or testing environment as this disables Device Has Stopped Responding to Polls alarms which disables the Alarm Impact calculations.