Spectroservers crashing at same time in a DSS environment
Article ID: 234830
Updated On:
Products
Issue/Introduction
Please consider this scenario:
It happens that sometimes a few/some landscapes. In a certain day out of the 10 landscapes crashed at the same time, with a coredump. Sometimes they can be restarted, sometimes not, and they crash again and again with a coredump.
To restart these, one has to stop the Main Location Server (MLS). When you restart the MLS, it also crashes with some dumps.
Here are the GDB (where) outputs from MLS and spectroservers:
MLS:
Program terminated with signal 6, Aborted.
#0 0x00007fdf4261a387 in raise () from /lib64/libc.so.6
#0 0x00007fdf4261a387 in raise () from /lib64/libc.so.6
#1 0x00007fdf4261ba78 in abort () from /lib64/libc.so.6
#2 0x00007fdf44756488 in Cs_out_of_memory() () from /opt/SPECTRUM/lib/libPort.so.1
#3 0x00007fdf431451ab in operator new(unsigned long) () from /lib64/libstdc++.so.6
#4 0x00007fdf43145289 in operator new[](unsigned long) () from /lib64/libstdc++.so.6
#5 0x00007fdf4bf3eeb8 in CsHPSESnmpv3Info::CsHPSESnmpv3Info(unsigned char const*, unsigned char const**, int, int) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#6 0x00007fdf4bf48700 in CsHPSnmpEngine::ResetAndLoadV3Profiles(CsBuffer const*, int, std::string) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#7 0x00007fdf4fcf1e98 in SnmpV3ProfileGlobalConfigIH::processProfileData(CsBuffer*) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1
#8 0x00007fdf4fcf37c4 in SnmpV3ProfileGlobalConfigIH::trig_model_create(CsModelHandle const&) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1
#9 0x00007fdf51e8bd84 in CsModel::trigger_create() () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#10 0x00007fdf51e8c60c in CsModel::CsModel(CsModelDesc const&, int, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#11 0x00007fdf51df60d5 in CsDupModel::CsDupModel(CsModelDesc const&, int, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#12 0x00007fdf51e7a3d3 in CsModelType::create(CsModelDesc const&, int, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#13 0x00007fdf51df0b6f in CsDupModelType::create(CsModelDesc const&, int, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#14 0x00007fdf51e548a6 in CsLandscape::load_models() () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#15 0x00007fdf51e59a59 in CsLandscape::CsLandscape(CsLscpeDesc const&, CsDbLandscape*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#16 0x00007fdf51efbff3 in VNMServer::vnm_start_up() () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#17 0x00007fdf51b11211 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1
#18 0x00007fdf4262c190 in ?? () from /lib64/libc.so.6
#19 0x0000000000000000 in ?? ()
Spectroservers (same output for all):
Program terminated with signal 11, Segmentation fault.
#0 0x00007fa061a18c6a in __strncpy_sse2_unaligned () from /lib64/libc.so.6
#0 0x00007fa061a18c6a in __strncpy_sse2_unaligned () from /lib64/libc.so.6
#1 0x00007fa06b2e461e in unpack_string(unsigned char const*, unsigned char const**) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#2 0x00007fa06b2d7be7 in CsHPSESnmpv3Info::CsHPSESnmpv3Info(unsigned char const*, unsigned char const**, int, int) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#3 0x00007fa06b2e1700 in CsHPSnmpEngine::ResetAndLoadV3Profiles(CsBuffer const*, int, std::string) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#4 0x00007fa06f08ae98 in SnmpV3ProfileGlobalConfigIH::processProfileData(CsBuffer*) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1
#5 0x00007fa06f08b45b in SnmpV3ProfileGlobalConfigIH::trig_attr_change(CsModelHandle const&, CsChangeNode const*) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1
#6 0x00007fa0711b7e9b in CsIHMTAttrWatcher::notify(CsIncidentParm const*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#7 0x00007fa07122cc3e in CsModel::react_to_change(CsVPList&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#8 0x00007fa071236d72 in CsModel::process_changes() () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#9 0x00007fa071252939 in CsNotificationMgr::process_queued_work(CsWorkNode const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#10 0x00007fa06c162512 in CsWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libwkmgr.so.1
#11 0x00007fa06c161a70 in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1
#12 0x00007fa070eaa211 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1
#13 0x00007fa0619c5190 in ?? () from /lib64/libc.so.6
#14 0x0000000000000000 in ?? ()
In this scenario, It was detected that 2 of the spectroservers were using different (older) SDC (secure domain connectors) versions than the others.
Environment
Release : 21.2
Component : Spectrum Core / SpectroSERVER
Cause
Spectrum maintains SNMPv3Profile attribute on GlobalConfig model which is common across all the SpectroSERVERs in the DSS i.e each SS access the same data.
Given there's a change in the profile attribute data in latest releases (21.2.4), trying to unpack an old attribute data (since it's not migrated) would result in a
SpectroSERVER crash probably in all landscapes.
Resolution
The problem was solved after upgrading the older SD connectors to the same version running in all the other Spectroservers.
If the problem still happens after having all the SDC's running in the same version, please contact the Broadcom Support Team.
Feedback
