One of our SpectroSERVERs is running slow and it outputting errors in the VNM.OUT: CORBA request timed out after 60000ms

Products

Spectrum Network Observability

Issue/Introduction

One of our SpectroSERVERs is running slow and it outputting errors in the VNM.OUT:

Jan 25 09:28:53 WARNING at ItcQdItem.cc(384): Request 1695653 not accepted after 60800ms, cs delta=0

Jan 25 09:28:53 WARNING at ItcQdItem.cc(384): Request 1695649 not accepted after 65478ms, cs delta=0

Saved compact diagnostic file to '/CA/spectrum/SS/support/SpectroSERVER_20240125_0928.dmp'

Jan 25 09:28:54 WARNING at ItcRequestParmsCorba.cc(277): CORBA request timed out after 60000ms.

Environment

Spectrum: All Supported Versions

Cause

The SpectroSERVER has embedded code to produce ‘mini-dump’ files if performance is impacted.

This is to help with troubleshooting.

…

From the .dmp file:

Thread 1 (Thread 0x7f3db069ecc0 (LWP 69892)):

#0 0x00007f3daff55818 in CsLandscape::find_models_by_derivation(CsFindSpec const&, CsULHashTable*, CsFindSpec::LogicalOp_e, CsULHashTable*, int, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#1 0x00007f3daff58a6e in CsLandscape::terminal_find_model_handles(CsFindSpec const*, CsULHashTable*, CsFindSpec::LogicalOp_e, CsError::CsError_e*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#2 0x00007f3daff5dc72 in CsLandscape::find_model_handles(CsFindSpec const*, CsError::CsError_e*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#3 0x00007f3daff5e3ad in CsLandscape::find_models(CsFindSpec const*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#4 0x00007f3daff5e4a4 in CsLandscape::find_models(CsFindSpec const*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#5 0x00007f3daffa5ee9 in CsModelDomainSrvc::get_model_desc_list(CsSecurityIf const&, char const*, char const*, CsULHashTable*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#6 0x00007f3daffa61e4 in CsModelDomainSrvc::getModelIDListByXmlSearchCriteria(CsCAttribute::CsCValue const&, CsSecurityIf const&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

Resolution

CsModelDomainSrvc::getModelIDListByXmlSearchCriteria

Points to a bad search overloading the SpectroSERVER.

This could be a global collection search, or an API call.

To check if it is a global collection search, you can query the DDMDB or reporting database in SpectroSERVER mysql.

…

cd /usr/Spectrum/mysql/bin/ (your path may vary)

./mysql --defaults-file=../my-spectrum.cnf -uroot -p

select * from ddmdb.event where type=0x1a102\G

Or if you have the reporting db:

select * from reporting.alarminfo where cause=0x1a102\G

If you get results, then it is likely a GC query that is the problem.

If no hits, it is likely an API call that is the problem.

Additional Information

Please reference the "Database Management" section of the documentation for additional information.