All of our SpectroSERVERs crashed due to an ineffiecent RESTful API search
Article ID: 8988
Updated On:
Products
Issue/Introduction
All of our SpectroSERVERs in our DSS environment crashed shortly after running a RESTful (Web Services) API search.
The stack output will look similar to this (with the key being CsFindSpec filling up the stack):
libPort.dll!malloc(unsigned __int64 cb=0x0000000000000040) Line 2420 + 0xb bytes C++
libPort.dll!operator new[](unsigned __int64 cb=0x0000000000000040) Line 2534 + 0xa bytes C++
libGlobl.dll!CsGHashTable::CsGHashTable(unsigned int (const void *)* hash_f=0x000007fee4fcdf60, int (const void *, const void *)* cmp_f=0x000007fee4fce2c0, unsigned char max_load_fac='', unsigned char min_load_fac='', unsigned int min_bucket_count=0x00000008) Line 227 + 0x2d bytes C++
libVPapi.dll!CsFindSpecAttrTerminal::CsFindSpecAttrTerminal(const CsFindSpecAttrTerminal & other={...}) Line 322 + 0x62 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2220 + 0x36 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 715 + 0x26 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C+
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::operator=(const CsFindSpec & rhs={...}) Line 2231 + 0x84 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & fs_attr_term={...}) Line 817 + 0xf bytes C++
libVPapi.dll!CsFindSpecExprTree::CsFindSpecExprTree(const CsFindSpec & in_left_expr={...}, CsFindSpec::LogicalOp_e log_op=OR, const CsFindSpec & in_right_expr={...}) Line 716 + 0x29 bytes C++
libVPapi.dll!CsFindSpec::CsFindSpec(const CsFindSpec & in_left={...}, CsFindSpec::LogicalOp_e logical_op=OR, const CsFindSpec & in_right={...}, CsFindSpec::debugMode_e eDebugFlag=eDebugNone) Line 1276 + 0x30 bytes C++
libmdlsvint.dll!SearchParser::reduceBinaryOperators(ControlNode* controlNode=0x000000000f680a00, SearchParser::ReductionDirection direction=eReducingUp) Line 536 + 0x53 bytes C++
libmdlsvint.dll!SearchParser::reduceStackToTree(SearchParser::ReductionDirection direction=eReducingUp) Line 124 + 0x14 bytes C++
libmdlsvint.dll!SearchParser::parseFilteredModels(_xmlNode * xmlPosition=0x000000000db58000) Line 2456 C++
libmdlsvint.dll!SearchParser::parseSearchCriteria(_xmlNode * xmlPosition=0x000000000db58528) Line 482 + 0x15 bytes C++
libmdlsvint.dll!SearchParser::parseDocument(_xmlNode * currentNode=0x000000000db58528) Line 3276 + 0xf bytes C++
libmdlsvint.dll!SearchParser::xmlStringToFindSpec(const char * xmlContents=0x0000000012781000) Line 3434 + 0x1d bytes C++
libmdlsvint.dll!IHParseSearchXml::trig_action(const CsModelHandle & mh={...}, const CsAction * action=0x000000000a6bf640) Line 154 + 0x15 bytes C++
libIHapi.dll!CsIHBase::trig_action(const CsModelHandle & mh={...}, const CsAction * action=0x000000000a6bf640, const CsConnHandle & __formal={...}) Line 2972 C++
libsskrnl.dll!CsModelType::trig_action(const CsModelHandle & mh={...}, const CsAction * action=0x000000000a6bf640, const CsVPConnHandle & conn_hndl={...}) Line 2142 + 0x49 bytes C++
libsskrnl.dll!CsModel::action(CsAction * action=0x000000000a6bf640, const CsVPConnHandle & conn_hndl={...}, const CsSecurityIf * user_security=0x0000000000000000) Line 883 + 0x33 bytes C++
libsskrnl.dll!CsModelIf::action(CsAction * action=0x000000000a6bf640, const CsVPConnHandle & conn_hndl={...}, const CsSecurityIf * user_security=0x0000000000000000) Line 1190 + 0x28 bytes C++
libIHapi.dll!CsVnmModelHandle::action(CsAction * action=0x000000000a6bf640, const CsConnHandle & conn_hndl={...}) Line 478 + 0x18 bytes C++
libIHapi.dll!CsVnmModelHandle::action(CsAction * model_action=0x000000000a6bf640) Line 449 C++
libsskrnl.dll!CsModelDomainSrvc::get_model_desc_list(const CsSecurityIf & user_security={...}, const char * xml=0x0000000008105000, const char * message_id=0x000007fee1088da0, CsULHashTable * search_subset=0x0000000000000000) Line 4496 + 0x13 bytes C++
libsskrnl.dll!CsModelDomainSrvc::getModelIDListByXmlSearchCriteria(const CsCAttribute::CsCValue & valueXml={...}, const CsSecurityIf & user_security={...}) Line 1807 + 0x28 bytes C++
libsskrnl.dll!CModelDomainItcM::processItcRequest(ITC_Request_Parms * itc_request=0x000000001447db80) Line 230 + 0x5f bytes C++
libitc.dll!ITC_Request_Parms::processRequest_IgnoreExceptions() Line 243 + 0x1c bytes C++
libitc.dll!ITC_Request_Parms_Corba::processRequest_TransferExceptions() Line 174 + 0xa bytes C++
libitc.dll!ItcWorkQueue::process_work_item(ItcQdItem * process_item=0x0000000014472cb0) Line 142 + 0x10 bytes C++
libitc.dll!ItcWorkQueue::process_queued_work(const CsWorkNode * work_node=0x0000000014472cb0) Line 116 C++
libitc.dll!ItcWorkQueue::process_work_node(CsWorkNode * work_node=0x0000000014472cb0) Line 166 C++
libwkmgr.dll!CsWorkScheduler::do_work() Line 393 C++
libmoot.dll!Thread::call_thread_function() Line 1283 C++
libmoot.dll!call_func(void * thread=0x0000000003025ec0) Line 83 C++
kernel32.dll!BaseFiberStart() + 0x1d bytes
Environment
Component:
Cause
The issue is caused by a stack overflow due to an inefficient search that is filling up the stack with CsFindSpec requests. Looking at the search used, we find multiple nested "OR" statements, which results in numerous CsFindSpec requests in the stack, which eventually overflows the stack and crashes the SpectroSERVER.
For example:
<?xml version="1.0" encoding="UTF-8"?>
<search-criteria xmlns="http://www.ca.com/spectrum/restful/schema/filter">
<filtered-models>
<or>
<or>
<equals>
<attribute id="0xffff0049">
<value>BLTOMDWU00W</value>
</attribute>
</equals>
<equals>
<attribute id="0xffff0049">
<value>BLTMMD9703W</value>
</attribute>
</equals>
</or>
<or>
<equals>
<attribute id="0xffff0049">
<value>BLTOMDPT00W</value>
</attribute>
</equals>
<equals>
<attribute id="0xffff0049">
<value>BLTRMDMY00W</value>
</attribute>
</equals>
</or>
<or>
<equals>
<attribute id="0xffff0049">
<value>BLTOMDIK02W</value>
</attribute>
</equals>
<equals>
<attribute id="0xffff0049">
<value>BLTOMDMW01W</value>
</attribute>
</equals>
</or>
</or>
</filtered-models>
</search-criteria>
Resolution
Modifying the search and replacing the nested "OR" statements with a single "OR", optimizes the search and reduces the number of CsFindSpec requests in the stack. Instead of sending a CsFindSpec for every "OR" in the search, we send a single CsFindSpec and do not overflow the stack.
For Example:
<?xml version="1.0" encoding="UTF-8"?>
<search-criteria xmlns="http://www.ca.com/spectrum/restful/schema/filter">
<filtered-models>
<or>
<equals>
<attribute id="0xffff0049">
<value>BLTOMDWU00W</value>
</attribute>
</equals>
<equals>
<attribute id="0xffff0049">
<value>BLTMMD9703W</value>
</attribute>
</equals>
<equals>
<attribute id="0xffff0049">
<value>BLTOMDPT00W</value>
</attribute>
</equals>
<equals>
<attribute id="0xffff0049">
<value>BLTRMDMY00W</value>
</attribute>
</equals>
<equals>
<attribute id="0xffff0049">
<value>BLTOMDIK02W</value>
</attribute>
</equals>
<equals>
<attribute id="0xffff0049">
<value>BLTOMDMW01W</value>
</attribute>
</equals>
</or>
</filtered-models>
</search-criteria>
Also, to help protect Spectrum form a stack overflow, due to ineffecient searches, Engineering suggested adding the following statement to the $SPECROOT/SS/.vnmrc:
stack_size_modifier=2
Provided there is enough available memory on the SpectroSERVER, setting the stack_size_modifier to 2 will help stabilize the SpectroSERVER. However, this does not provent the SpectroSERVER from crashing all together, if someone runs a poorly constructed search, or really large search request. It simply increases the stack size to help us handle larger stacks. If someone comes in with a doubly big RESTful query, we could run into the same problem.
Engineering's testing shows when the stack_size_modifier=2, we see Work Thread Pooled go from the overflow condition to around 84%. This change gives Spectrum room for a ~16% bigger query. If the SpectroSERVER has more available memory, a setting of stack_size_modifier=4 is a better option.
NOTES: Setting stack_size_modifier=2 at worst case will increase the SpectroSERVER memory footprint by 256 MB (64k*4000 thread)
Setting stack_size_modifier=4 at worst case will increase the SpectroSERVER memory footprint by 512 MB.
Feedback
