Non-MLS SpectroSERVER's are crashing after an upgrade to 10.3.2 and 10.4.x

book

Article ID: 139955

calendar_today

Updated On:

Products

CA Spectrum CA eHealth

Issue/Introduction

After upgrading our MLS to Spectrum 10.3.2 or 10.4.x and starting it up, all non-mls servers are crashing. The core files are showing the following stack:

Program terminated with signal 11, Segmentation fault.
#0 0x00007fcb3707cd90 in unpack_ulong(unsigned char const*, unsigned char const**) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#0 0x00007fcb3707cd90 in unpack_ulong(unsigned char const*, unsigned char const**) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#1 0x00007fcb3707cf33 in unpack_string(unsigned char const*, unsigned char const**) () from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#2 0x00007fcb370568db in CsHPSESnmpv3Info::CsHPSESnmpv3Info(unsigned char const*, unsigned char const**, int) ()
from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#3 0x00007fcb37062368 in CsHPSnmpEngine::ResetAndLoadV3Profiles(CsBuffer const*, int, std::string) ()
from /opt/SPECTRUM/lib/../SS/libhpse.so.1
#4 0x00007fcb3ad96f59 in SnmpV3ProfileGlobalConfigIH::processProfileData(CsBuffer*) ()
from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1
#5 0x00007fcb3ad9708b in SnmpV3ProfileGlobalConfigIH::trig_attr_change(CsModelHandle const&, CsChangeNode const*) ()
from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1
#6 0x00007fcb3cbf8e87 in CsIHMTAttrWatcher::notify(CsIncidentParm const*, CsSecurityIf const*) ()
from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#7 0x00007fcb3cc50c8d in CsModel::react_to_change(CsVPList&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#8 0x00007fcb3cc5ac47 in CsModel::process_changes() () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#9 0x00007fcb3cc8e777 in CsNotificationMgr::process_queued_work(CsWorkNode const*) ()
from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#10 0x00007fcb37eac452 in CsWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libwkmgr.so.1
#11 0x00007fcb37eabb33 in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1
#12 0x00007fcb3c8f2161 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1
#13 0x00007fcb3113f0d0 in ?? () from /lib64/libc.so.6
#14 0x0000000000000000 in ?? ()

Cause

SNMPv3 Profile Migration between primary and secondary SpectroSERVERs

Environment

Release : 10.3.2 & 10.4.x

Component : Spectrum Core / SpectroSERVER

Resolution

SE has validated the below conditions which causes the crash in a fault tolerant environment.

- All primary SpectroSERVER's are upgraded and started at the same time.
- When they are all started at the same time, the primary MLS is not fully ready yet, and during this brief window, the non-MLS systems then connect to the secondary MLS running the older verison of Spectrum.
- When this connection happens, it syncs the v3 profile migration and writes to the SNMPv3profileattr on the GlobalConfig model. During the migration it sees there is a mismatch between the versions and then crashes all of the non-MLS systems.
- Also, if a user were to try and create a v3 profile during this time, it will crash the system as well.

Since the v3 code has changed dramatically since 10.3.2, they are going to update the documentation to explicitly state that the following is the only supported upgrade method:.

1. Bring down all primary SpectroSERVER's.
2. Upgrade all primary SpectroSERVER's.
3. Start only the primary MLS first, and make sure it is 100% activated and is running properly.
4. Start all other non-MLS primary SpectroSERVER's.
5. Confirm that the primaries are connected properly by verifying the landscape map using MapUpdate.
6. Run through the same procedure on the secondary systems.

NOTE - Do not create any V3 profiles during the upgrade process, It can crash the servers.