SpectroServer crashing regularly.
Article ID: 259683
Updated On:
Products
Issue/Introduction
The secondary SpectroServer has been crashing regularly.
core.30291:
[?1034h(gdb) where
#0 0x00007fe095a45079 in vfprintf () from /lib64/libc.so.6
#1 0x00007fe095a70179 in vsnprintf () from /lib64/libc.so.6
#2 0x00007fe09f3ae8ce in LCL_FormatVA () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#3 0x00007fe09f3aed9b in LCL_SprintfVA () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#4 0x00007fe09f3b3c0d in _LOG_MessageVA () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#5 0x00007fe09f3b3fd5 in LOG_Message () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#6 0x00007fe09f3a4742 in cs_snmpv3_process_incoming.isra.0 () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#7 0x00007fe09f3a52fb in cs_snmp_parse_v3_pdu () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#8 0x00007fe09f3946d5 in cs_snmp_parse_pdu () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#9 0x00007fe09f390fee in SNMP_TrapResponseUnpack () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#10 0x00007fe09f357110 in HideSnmpTrapInternals::GetTrapResponse(int, _SNMP_PACKET_struct_tag**, std::string&) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#11 0x00007fe09f35727a in CsHPSETrapReceiver::DoTask() () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#12 0x00007fe09f3558f6 in CsHPSETask::threadRun(void*) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#13 0x00007fe095dcdea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007fe095af6b0d in clone () from /lib64/libc.so.6
core.22051:
[?1034h(gdb) where
#0 0x00007f28ddc13079 in vfprintf () from /lib64/libc.so.6
#1 0x00007f28ddc3e179 in vsnprintf () from /lib64/libc.so.6
#2 0x00007f28e757c8ce in LCL_FormatVA () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#3 0x00007f28e757cd9b in LCL_SprintfVA () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#4 0x00007f28e7581c0d in _LOG_MessageVA () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#5 0x00007f28e7581fd5 in LOG_Message () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#6 0x00007f28e7572742 in cs_snmpv3_process_incoming.isra.0 () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#7 0x00007f28e75732fb in cs_snmp_parse_v3_pdu () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#8 0x00007f28e75626d5 in cs_snmp_parse_pdu () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#9 0x00007f28e755efee in SNMP_TrapResponseUnpack () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#10 0x00007f28e7525110 in HideSnmpTrapInternals::GetTrapResponse(int, _SNMP_PACKET_struct_tag**, std::string&) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#11 0x00007f28e752527a in CsHPSETrapReceiver::DoTask() () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#12 0x00007f28e75238f6 in CsHPSETask::threadRun(void*) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#13 0x00007f28ddf9bea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f28ddcc4b0d in clone () from /lib64/libc.so.6
---------------
Environment
Release : 22.2
Cause
There is an attribute (DebugSnmpStack = 0x000130f8) on VNM model, which was written to the dB, on the Secondary.
It was set TRUE in the catalog for the secondary, which was causing the issue. Run the below to verify.
./show attributes mh=<vnm_mh> | grep "DebugSnmpStack"
Resolution
Disabling the debug stopped the crashes as follows:
update action=0x10315 mh=<VNM_MH> # to Disable Tracing
and
./update mh=<vnm_mh> attr=0x130f8,val=FALSE
Additional Information
This debug is disabled by default and will cause porblems unless enabled. Engineering are working on fixing the debug, which will be added to a future release, most likely 22.2.8 (as of Feb 2023)
If the SS is not dumping cores, please set the following on the primary server to ensure it dumps a core in future.
ulimit -c unlimited (both as root and spectrum install_owner)
Feedback
