"CORBA request timed out after 60000ms, possible performance problem" errors seen in CA Spectrum Control Panel and VNM.OUT

Products

• Spectrum
• Network Observability

Issue/Introduction

Description:

The following output is seen in the Spectrum Control Panel and <SPECROOT>/SS/VNM.OUT file:

ERROR at ItcRequestParmsCorba.cc(92): CORBA request timed out after 60000ms, possible performance problem.

This indicates there was a delay with information being processed in the SpectroSERVER. The output in the <SPECROOT>/SS/VNM.OUT file will look like this:

Dec 20 08:26:46 ERROR at ItcQdItem.cc(219): elapsed wait time = 60000 milliseconds.
Dec 20 08:26:46 ERROR at ItcQdItem.cc(223): context switch delta = 0
Dec 20 08:26:46 ERROR at ItcRequestParmsCorba.cc(92): CORBA request timed out after 60000ms, possible performance problem.
Saved compact diagnostic file to 'C:\win32app\Spectrum\SS\support\SpectroSERVER_20111220_0826.dmp'
Dec 20 08:27:34 ERROR at ItcQdItem.cc(219): elapsed wait time = 60000 milliseconds.
Dec 20 08:27:34 ERROR at ItcQdItem.cc(223): context switch delta = 0
Dec 20 08:27:34 ERROR at ItcRequestParmsCorba.cc(92): CORBA request timed out after 60000ms, possible performance problem.
Dec 20 08:27:39 ERROR at ItcQdItem.cc(219): elapsed wait time = 60000 milliseconds.
Dec 20 08:27:39 ERROR at ItcQdItem.cc(223): context switch delta = 0
Dec 20 08:27:39 ERROR at ItcRequestParmsCorba.cc(92): CORBA request timed out after 60000ms, possible performance problem.
Dec 20 08:46:55 ERROR at ItcQdItem.cc(219): elapsed wait time = 60000 milliseconds.
Dec 20 08:46:55 ERROR at ItcQdItem.cc(223): context switch delta = 0
Dec 20 08:46:55 ERROR at ItcRequestParmsCorba.cc(92): CORBA request timed out after 60000ms, possible performance problem.
Dec 20 08:46:55 ERROR at ItcQdItem.cc(219): elapsed wait time = 60000 milliseconds.
Dec 20 08:46:55 ERROR at ItcQdItem.cc(223): context switch delta = 0
Saved compact diagnostic file to 'C:\win32app\Spectrum\SS\support\SpectroSERVER_20111220_0846.dmp'

 

Environment

Release: All Supported Spectrum Releases
Component: SPCCSS - SpectroSERVER Core

Resolution

Out of the box, Spectrum will automatically create stack dumps if a delay occurs within the SpectroSERVER process for more than 3 minutes. This debug is enabled by default via the following setting in the <SPECROOT>/SS/.vnmrc file (the entry is not there by default but is enabled by default):

enable_itc_moot_perf_debug=true

Please note that this debug does not work on Linux if the pstack utility is not installed.

It is not recommended to do so, but if you need to disable this debug because the .moot.trace file is over 1GB in size, set the value to false and then stop/restart the SpectroSERVER:

enable_itc_moot_perf_debug=false

If you are seeing that the *.dmp files are being created, please zip up the entire <SPECROOT>/SS/support directory. Open a case with Spectrum Support and provide the zip file as well as a copy of the $SPECROOT/Install-Tools/.history file and the <SPECROOT>/SS/.moot.trace file.

CA Spectrum Support will analyze the stack dumps and the version of Spectrum you are running and provide you with an analysis and recommendation to resolve or further debug.

PLEASE NOTE: Depending on the stack information, further performance debugging may be needed.