almost all spectroSERVERs crashed at same time in a DSS environment
Article ID: 262686
Updated On:
Products
Issue/Introduction
Almost all spectroSERVERs crashed at same time in a DSS environment, with the following stack
Core was generated by `/opt/Spectrum/SS/
Program terminated with signal 11, Segmentation fault.
#0 0x00007f59063a7320 in CsBuffer::makestr(int, char) const () from /opt/SPECTRUM/lib/libGlobl.so.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 nss-softokn-freebl-3.79.0-4.
(gdb) where
#0 0x00007f59063a7320 in CsBuffer::makestr(int, char) const () from /opt/SPECTRUM/lib/libGlobl.so.
#1 0x00007f59119e7150 in CsLandscape::search_keyed_
#2 0x00007f59119e77cd in CsLandscape::terminal_find_
#3 0x00007f59119ec701 in CsLandscape::find_model_
#4 0x00007f59119ed17d in CsLandscape::find_models(
#5 0x00007f59119ed274 in CsLandscape::find_models(
#6 0x00007f5911a34cb9 in CsModelDomainSrvc::get_model_
#7 0x00007f5911a34fb4 in CsModelDomainSrvc::getModelIDL
#8 0x00007f5911966b7f in CModelDomainItcM::
#9 0x00007f590efb060b in ITC_Request_Parms_Corba::
#10 0x00007f590efaf9c7 in ItcWorkQueue::process_work_
#11 0x00007f590efaf802 in ItcWorkQueue::process_work_
#12 0x00007f590c93da70 in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.
#13 0x00007f5911695211 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1
#14 0x00007f59021a0190 in ?? () from /lib64/libc.so.6
#15 0x0000000000000000 in ?? ()
Environment
Release : 22.2.x
Cause
The crash was caused by a rest API call which specified the DOMAIN_ID attribute in his body.
<?xml version="1.0" encoding="UTF-8"?>
<!--
CA Technologies, Inc.
One CA Plaza
Islandia, NY 11749 USA
Copyright (c) 2014 CA Technologies, Inc.
All rights reserved.
IN NO EVENT SHALL CA TECHNOLOGIES INCORPORATED BE LIABLE FOR
ANY INCIDENTAL, INDIRECT, SPECIAL, OR CONSEQUENTIAL DAMAGES
WHATSOEVER (INCLUDING BUT NOT LIMITED TO LOST PROFITS) ARISING OUT
OF OR RELATED TO THIS SOFTWARE, EVEN IF CA TECHNOLOGIES INCORPORATED
HAS BEEN ADVISED OF, KNOWN, OR SHOULD HAVE KNOWN, THE POSSIBILITY OF
SUCH DAMAGES.
-->
<rs:model-request throttlesize="2000000000"
xmlns:rs="http://www.ca.com/spectrum/restful/schema/request"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.ca.com/spectrum/restful/schema/request ../../../xsd/Request.xsd ">
<rs:target-models>
<rs:models-search>
<rs:search-criteria
xmlns="http://www.ca.com/spectrum/restful/schema/filter">
<devices-only-search/>
<filtered-models>
<and>
<has-substring-ignore-case>
<attribute id="0x12e4d"><value>false</value></attribute>
</has-substring-ignore-case>
<has-substring-ignore-case>
<attribute id="0x129ac"><value>0x87000000</value></attribute>
</has-substring-ignore-case>
</and>
</filtered-models>
</rs:search-criteria>
</rs:models-search>
</rs:target-models>
<rs:requested-attribute id="0x1006e" /> <!-- model_name -->
<rs:requested-attribute id="0x6250007" /> <!-- group_2ndl -->
<rs:requested-attribute id="0x12bfc" /> <!-- ci_id -->
<rs:requested-attribute id="0x10024" /> <!-- community_string -->
<rs:requested-attribute id="0x1027f" /> <!-- network_address -->
<rs:requested-attribute id="0x10000" /> <!-- model_type -->
<rs:requested-attribute id="0x10023" /> <!-- agent_port -->
<rs:requested-attribute id="0x12d83" /> <!-- secdomain_ipname -->
<rs:requested-attribute id="0x12c04" /> <!-- secdomain_name -->
</rs:model-request>
and that resulted in an
Error.NoAvailableDomains
with the following exception in the OC tomcat log
Mar 24, 2023 12:21:24.798 (http-nio-8080-exec-12) (RESTFULWEBSERVICES) - (DEBUG) - in _POST_getModelsByXML starting with uri http://xxxxxx:8080/spectrum/restful/models?throttlesize=200&landscape=0x20000000
Mar 24, 2023 12:21:26.942 - SearchTask: Error searching domain xxxxx.yyyyy: org.omg.CORBA.TRANSIENT: vmcid: 0x0 minor code: 0 completed: No at com.inprise.vbroker.orb.DelegateImpl.verifyConnection(Unknown Source) at com.inprise.vbroker.orb.DelegateImpl.is_local(Unknown Source) at org.omg.CORBA.portable.ObjectImpl._is_local(ObjectImpl.java:372) at com.aprisma.spectrum.core.idl._CsCModelDomainStub.getModelIDListByXmlSearchCriteria(_CsCModelDomainStub.java:7667) at com.aprisma.spectrum.app.web.search.model.XMLSearchExpression.doOperation(XMLSearchExpression.java:268) at com.aprisma.spectrum.app.web.search.model.SearchExpression.getModelHandleList(SearchExpression.java:187) at com.aprisma.spectrum.app.web.search.model.SearchModel$SearchTask.performTask(SearchModel.java:96) at com.aprisma.spectrum.app.web.search.model.SearchModel.doSearch(SearchModel.java:389) at com.aprisma.spectrum.app.web.search.model.SearchModel.getModelHandleList(SearchModel.java:218)
......
NOTE: The above debug is only seen if the "RESTful Web Services" debug is enabled in the OneClick Administration -> Debugging -> Web Server Debug Page (Runtime) web pages.
Resolution
Remove the DOMAIN_ID attribute
<has-substring-ignore-case>
<attribute id="0x129ac"><value>0x87000000</value></attribute>
</has-substring-ignore-case>
from the body of rest call.
If you need to filter the models of a specific landscape specify the landscape id as follows:
<rs:model-request throttlesize="2000000000"
xmlns:rs="http://www.ca.com/spectrum/restful/schema/request"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.ca.com/spectrum/restful/schema/request ../../../xsd/Request.xsd ">
<rs:landscape id="0x87000000" />
<rs:target-models>
<rs:models-search>
<rs:search-criteria
.....
Additional Information
User Story US793644 Proactively prevent SS crash on invalid REST query (32927148 and 33392695) already exists to address this in future.
Feedback
