SpectroServer crashes several times since upgrading to 21.2.8
Article ID: 240073
Updated On:
Products
Issue/Introduction
After upgrading the SpectroSERVERs and SDC´s to 21.2.8 we notice that the SpectroSERVERs are crashing about once per week.
The GDB trace from the core file looks the same as it is crashing with CORBA Visibroker in stack.
#0 0x00007f8a17f92272 in CORBA_MarshalInBuffer::get_int(int&) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#1 0x00007f8a17f94bab in CORBA_MarshalInBuffer::operator>>(int&) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#2 0x00007f89dce72de3 in operator>>(VISistream&, RTMDomain::RTM_Threshold&) () from /usr/Spectrum/SS/Modules/librtms.so.1
#3 0x00007f89dce7389a in operator>>(VISistream&, RTMDomain::RTM_Threshold_List&) () from /usr/Spectrum/SS/Modules/librtms.so.1
#4 0x00007f89dce73d52 in operator>>(VISistream&, RTMDomain::RTM_CusThreshold_Config&) () from /usr/Spectrum/SS/Modules/librtms.so.1
#5 0x00007f89dce73ff4 in operator>>(VISistream&, RTMDomain::RTM_CusThreshold_Config_var&) () from /usr/Spectrum/SS/Modules/librtms.so.1
#6 0x00007f89dce8858d in POA_RTMDomain::_configureCusThreshold(void*, CORBA_MarshalInBuffer&, char const*, VISReplyHandler&) () from /usr/Spectrum/SS/Modules/librtms.so.1
#7 0x00007f8a17e23ce1 in VISSkelInfo::execute(VISGIOPRequest const&, CORBA_MarshalInBuffer&, VISReplyHandler&) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#8 0x00007f8a17e3697c in VISGIOPProtocolAdapter::_do_request(VISGIOPRequest*, VISGlobalTable*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#9 0x00007f8a17e35946 in VISGIOPProtocolAdapter::dispatchMessage(ProtocolEngine::Message*, VISThreadSpecificData*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#10 0x00007f8a17f12860 in VISThreadPoolDispatcher::do_work(ProtocolEngine::ProtocolAdapter*, int&, VISThreadSpecificData*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#11 0x00007f8a17f12a9b in VISDispatchWorker::do_work(VISThreadSpecificData*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#12 0x00007f8a17f0723f in VISThreadPool::_perform_task(VISTPoolWorker*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#13 0x00007f8a17f0759a in VISTPoolWorker::begin() () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#14 0x00007f8a178e7f13 in VISThread::_start(void*) () from /opt/SPECTRUM/lib/libvport64_r.so
#15 0x00007f8a16a7fea5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f8a167a8b0d in clone () from /lib64/libc.so.6
Environment
Release : 21.2.8
Component : Spectrum Core / SpectroSERVER
Cause
Customer had 2 OneClick servers that were not upgraded yet, due to using Citrix and the authentication has changed
Resolution
SE has confirmed that the crash was due to mismatched versions between the 2 non upgraded OneClick servers and the rest of the upgraded environments.
After customer has upgraded the remaining OneClick servers, the crashes are no longer seen.
Additional Information
While we have never supported a mixed version Spectrum environment, sometimes in the past it would work even though it was not supported.
However going from an older version such as 10.4.2.0 to the newer versions of Spectrum such as 21.2.8 can and will cause crashes and instability in your Spectrum environment.
Please keep this in mind when you plan your upgrade to upgrade all of your servers together as recommended by the documentation and support.
Feedback
