One of our SpectroSERVERs is running slow and it outputting errors in the VNM.OUT: CORBA request timed out after 60000ms

Products

• Spectrum
• Network Observability

Issue/Introduction

SpectroSERVER is running slow and is outputting the following messages in the $SPECROOT/SS/VNM.OUT log:

Jan 25 09:28:53 WARNING at ItcQdItem.cc(384): Request 1695653 not accepted after 60800ms, cs delta=0
Jan 25 09:28:53 WARNING at ItcQdItem.cc(384): Request 1695649 not accepted after 65478ms, cs delta=0
Saved compact diagnostic file to '/CA/spectrum/SS/support/SpectroSERVER_20240125_0928.dmp'
Jan 25 09:28:54 WARNING at ItcRequestParmsCorba.cc(277): CORBA request timed out after 60000ms.
Jan 25 09:28:54 WARNING at ItcRequestParmsCorba.cc(277): CORBA request timed out after 60000ms.

Environment

DX NetOps Spectrum: All currently supported releases

Cause

The SpectroSERVER has embedded code to produce ‘mini-dump’ files if performance is impacted.  This is to help with troubleshooting.

…

From the .dmp file the following is seen:

Thread 1 (Thread 0x7f3db069ecc0 (LWP 69892)):

#0  0x00007f3daff55818 in CsLandscape::find_models_by_derivation(CsFindSpec const&, CsULHashTable*, CsFindSpec::LogicalOp_e, CsULHashTable*, int, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#1  0x00007f3daff58a6e in CsLandscape::terminal_find_model_handles(CsFindSpec const*, CsULHashTable*, CsFindSpec::LogicalOp_e, CsError::CsError_e*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#2  0x00007f3daff5dc72 in CsLandscape::find_model_handles(CsFindSpec const*, CsError::CsError_e*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#3  0x00007f3daff5e3ad in CsLandscape::find_models(CsFindSpec const*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#4  0x00007f3daff5e4a4 in CsLandscape::find_models(CsFindSpec const*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#5  0x00007f3daffa5ee9 in CsModelDomainSrvc::get_model_desc_list(CsSecurityIf const&, char const*, char const*, CsULHashTable*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#6  0x00007f3daffa61e4 in CsModelDomainSrvc::getModelIDListByXmlSearchCriteria(CsCAttribute::CsCValue const&, CsSecurityIf const&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

Resolution

CsModelDomainSrvc::getModelIDListByXmlSearchCriteria

The above code reference points to a bad search overloading the SpectroSERVER.  This could be a global collection search, or an API call.

To check if it is a global collection search, you can query the DDMDB or reporting database in SpectroSERVER mysql as per the following:

 

1. cd /usr/Spectrum/mysql/bin/ (your path may vary)
   
   ./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWORD>
   
   
2. The Event type/code 0x1a102 is an event and alarm cause code used to identify performance-impacting searches, typically associated with Global Collection (GC)queries. We can search for this in the MySQL DB by running the following at the MySQL prompt;
   
   select * from ddmdb.event where type=0x1a102\G


3. On the OneClick server, if you have the reporting db (SRM), login to mysql as per step 1 and run;
   
   select * from reporting.alarminfo where cause=0x1a102\G

 

If you get results, then it is likely a GC query that is the problem.

If no hits, it is likely an API call that is the problem.

Additional Information

Please reference the "Database Management" section of the documentation for additional information:

TechDocs : DX NetOps Spectrum 25.4 : Database Management