Multiple Production SS Crashes when REST requested data from modeltypes that did not exist
search cancel

Multiple Production SS Crashes when REST requested data from modeltypes that did not exist

book

Article ID: 194735

calendar_today

Updated On:

Products

CA Spectrum CA eHealth

Issue/Introduction

Several primary SpectroSERVERs (including MLS) had crashed and failed over to the Secondary systems. When reviewing the stack, it initially appears that it was a REST query that caused it as noted by this:  getModelIDListByXmlSearchCriteria

However, we just made some modeltype changes.

(gdb) where
#0  0x00007f697755426d in CsULHashTable::get_node(unsigned int) const () from /opt/SPECTRUM/lib/libGlobl.so.1
#1  0x00007f69775542b1 in CsULHashTable::get(unsigned int) () from /opt/SPECTRUM/lib/libGlobl.so.1
#2  0x00007f697f13fbec in CsAttributeTable::get(unsigned int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#3  0x00007f697f1c6ed7 in CsModelType::flush_attribute(CsModelType*, unsigned int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#4  0x00007f697f1c6eef in CsModelType::flush_attribute(CsModelType*, unsigned int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#5  0x00007f697f1c7068 in CsModelType::set_attr_val(unsigned int, void const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#6  0x00007f697f1c7274 in CsModelType::write(CsAttrWriteReqList*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#7  0x00007f697e0c7bc6 in CsVnmMTypeHandle::write(CsAttrWriteReqList*) const () from /opt/SPECTRUM/lib/../SS/libIHapi.so.1
#8  0x00007f697f23e295 in KeyedAttributeCacheIH::add(CsAttrDesc const&, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#9  0x00007f697f1b1f45 in CsLandscape::load_keys(CsAttrDesc*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#10 0x00007f697f1b51c3 in CsLandscape::get_find_model_list(CsFindSpec const*, CsULHashTable*, CsFindSpec::LogicalOp_e, int, std::vector<CsModelHandle, std::allocator<CsModelHandle> >&, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#11 0x00007f697f1b5455 in CsLandscape::search_keyed_models(CsFindSpec const*, CsULHashTable*, CsFindSpec::LogicalOp_e, CsULHashTable*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#12 0x00007f697f1b610d in CsLandscape::terminal_find_model_handles(CsFindSpec const*, CsULHashTable*, CsFindSpec::LogicalOp_e, CsError::CsError_e*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#13 0x00007f697f1bb332 in CsLandscape::find_model_handles(CsFindSpec const*, CsError::CsError_e*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#14 0x00007f697f1bba6d in CsLandscape::find_models(CsFindSpec const*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#15 0x00007f697f1bbb64 in CsLandscape::find_models(CsFindSpec const*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#16 0x00007f697f202b89 in CsModelDomainSrvc::get_model_desc_list(CsSecurityIf const&, char const*, char const*, CsULHashTable*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#17 0x00007f697f202e84 in CsModelDomainSrvc::getModelIDListByXmlSearchCriteria(CsCAttribute::CsCValue const&, CsSecurityIf const&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#18 0x00007f697f13555f in CModelDomainItcM::processItcRequest(ITC_Request_Parms*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#19 0x00007f697c7935fb in ITC_Request_Parms_Corba::processRequest_TransferExceptions() () from /opt/SPECTRUM/lib/libitc.so.1
#20 0x00007f697c7929b7 in ItcWorkQueue::process_work_item(ItcQdItem*) () from /opt/SPECTRUM/lib/libitc.so.1
#21 0x00007f697c792802 in ItcWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libitc.so.1
#22 0x00007f697a128a70 in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1
#23 0x00007f697ee64211 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1
#24 0x00007f6972ef6d40 in ?? () from /lib64/libc.so.6
#25 0x0000000000000000 in ?? ()
(gdb)

Environment

Release : 10.4.1

Component : Spectrum Core / SpectroSERVER

Cause

The modeltype changes in the catalog were not loaded on all SpectroSERVERS.  When the REST function was executed, it crashed the servers that did not have the modeltype change.

Resolution

Save the catalog on the SS the modeltype changes were done on, and load it in on the rest of the SpectroSERVERs.

1.  Save the catalog while the SS is stopped (or you can run the online backup on the SS you made the changes on and use the backup file).

a.  Navigate to the $SPECROOT/SS directory and run the following command:

../SS-Tools/SSdbsave -c SSdb_catalog_only.SSdb

2.  Transfer either the backup file from the online backup or the SSdb_catalog_only.SSdb file to all other SS.

3.  Stop the SS to load the catalog.

4.  Load the catalog by navigating to the $SPECROOT/SS directory and run the following command:

../SS-Tools/SSdbload -c SSdb_catalog_only.SSdb

5.  Save the database with this command (or use the Spectrum Control Panel):

../SS-Tools/SSdbsave -cm SSdb_full_save.SSdb

6. Start the SpectroSERVER

Powered by Wolken Software