All SpectroSERVERs crashed after running REST query
search cancel

All SpectroSERVERs crashed after running REST query

book

Article ID: 132346

calendar_today

Updated On:

Products

CA Spectrum

Issue/Introduction

All SpectroSERVERs crashed after running RESTful query

STACK_TEXT: 
00000000`9aae3fd0 000007fe`edb9859d : 00000000`00000000 00000000`00000000 ffffffff`fffffffe 00000000`00000000 : mtsfm64!_malloc__+0x1b 
00000000`9aae4050 000007fe`eccd81aa : 00000000`0000001f 00000000`e83bc960 00000002`6ab42c80 00000000`00000000 : libPort!malloc+0x8d [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\port.a\src\memtrace.cc @ 2420] 
00000000`9aae4080 000007fe`eccfb65d : 00000002`bcf7f820 00000000`00000000 00000002`bcf7f820 000007fe`edb9859d : libVPapi!attrdup+0x26 [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\vpapi.a\src\csattrfcn.cc @ 220] 
00000000`9aae40b0 000007fe`eccfbeff : 00000002`bcf7f820 00000002`bcf7f820 00000000`00000000 00000000`00000000 : libVPapi!CsFindSpecAttrTerminal::CsFindSpecAttrTerminal+0x109 [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\vpapi.a\src\csfindspec.cc @ 338] 
00000000`9aae4100 000007fe`eccfadc6 : 00000000`e83bcf00 000007fe`edb9859d 00000002`bcf7f820 00000000`00000000 : libVPapi!CsFindSpec::operator=+0x1bb [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\vpapi.a\src\csfindspec.cc @ 2220] 
00000000`9aae41a0 000007fe`eccfba4e : 00000000`e83bc960 00000000`00000000 00000002`bcf7f4d8 00000000`00000000 : libVPapi!CsFindSpec::CsFindSpec+0x3e [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\vpapi.a\src\csfindspec.cc @ 817] 
00000000`9aae41e0 000007fe`eccfbeba : 00000000`e83bc960 00000002`bcf7fc30 00000000`e83bcc80 00000000`00000000 : libVPapi!CsFindSpecExprTree::CsFindSpecExprTree+0x4a [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\vpapi.a\src\csfindspec.cc @ 715] 
00000000`9aae4220 000007fe`eccfadc6 : 00000001`2cc2eca8 000007fe`edb9859d 00000002`bcf7fc30 00000000`00000000 : libVPapi!CsFindSpec::operator=+0x176 [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\vpapi.a\src\csfindspec.cc @ 2231] 
00000000`9aae42c0 000007fe`eccfba75 : 00000000`e83bcc80 00000000`00000000 00000002`bcf7fc30 00000000`00000000 : libVPapi!CsFindSpec::CsFindSpec+0x3e [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\vpapi.a\src\csfindspec.cc @ 817] 
00000000`9aae4300 000007fe`eccfbeba : 00000000`e83bcc80 00000002`bcf7f690 00000000`e83bc460 00000000`00000000 : libVPapi!CsFindSpecExprTree::CsFindSpecExprTree+0x71 [d:\spectrum\10.02.03\cm\windows\10.02.03.00.107\vpapi.a\src\csfindspec.cc @ 716] 

Environment

CA Spectrum 10.2.3

Cause

We were able to identify the problem, it is because a huge REST API request which caused one of the SS threads to run out of stack memory while handling the REST request 

Spectrum 10.3.2 has a prevent fix to avoid this kind the situation happens for limiting the max number of nodes to 150 in search query before it is sent to SS 

Resolution

Contact the Broadcom support and request the Spectrum_10.02.03.PTF_10.2.393a PTF patch.

Additional Information

As you know that, SS fails to handle the HUGE REST query when it receives from OC Server and also generates a big core file while processing that huge REST query.

In 393a patch, we have restricted the number of nodes which are present in REST search query to avoid the SS crash.

It's always good to prepare optimal search query instead of giving all IP addresses in the search list.

Kindly suggest customer to use "By IP Address Range" filter or giving less number of IPs with "By IP Address" filter to process the query.

 

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/spectrum/10-3-2/release-information/bi-monthly-patches-bmps-on-10-3-2.html

DE417938
Symptom: SpectroSERVER Crashes when huge REST query receives from OneClick server.
Resolution: SpectroSERVER does not Crashes and shows an Error.SearchQueryCombinationsExceedStackProcessingSize error to OneClick server when huge REST query receives.

Powered by Wolken Software