SpectroSERVER performance issue with CORBA unregister_watches

Products

• CA Spectrum

Issue/Introduction

The SpectroSERVER experiences a performance issue with CORBA while performing "unregister_watches" actions.  This causes the OneClick clients to turn red.  The VNM.OUT will show performance dumps being created along with timeout messages:

Apr 09 22:35:33 WARNING at ItcQdItem.cc(351): Request 611830696 not accepted after 120006ms, cs delta=0
Apr 09 22:35:33 WARNING at ItcRequestParmsCorba.cc(277): CORBA request timed out after 60000ms.
Apr 09 22:35:33 WARNING at ItcQdItem.cc(351): Request 611837298 not accepted after 60001ms, cs delta=0
Saved compact diagnostic file to 'D:\win32app\Spectrum\SS\support\SpectroSERVER_20190409_2235.dmp'
Apr 09 22:35:33 WARNING at ItcRequestParmsCorba.cc(277): CORBA request timed out after 60000ms.
Apr 09 22:35:34 WARNING at ItcQdItem.cc(351): Request 611831719 not accepted after 120009ms, cs delta=0
Apr 09 22:35:34 WARNING at ItcRequestParmsCorba.cc(277): CORBA request timed out after 60000ms.

When reviewing the stack in the dump file, the "unregister_watches" action is shown:

RetAddr : Args to Child : Call Site 
00007ffe`07b74c9b : 000000ef`00ef53a8 000000ef`00000000 000000ee`e675fd70 000000ee`0082a840 : libsskrnl!CsModel::unreg_attr_watch+0xa3 [d:\spectrum\sunshine\cm\windows\10.03.00.00.341\sscor.a\lscpe\src\csmodel4.cc @ 1905] 
00007ffe`07ac9fe9 : 00000000`001b0756 000000ef`582f7348 00000000`001b0756 000000ef`58208c00 : libsskrnl!CsModelDomainSrvc::unregister_watches+0x3a7 [d:\spectrum\sunshine\cm\windows\10.03.00.00.341\sscor.a\corba\src\csmodeldomsrvc.cc @ 782] 
*** WARNING: Unable to verify checksum for libitc.dll

Environment

Release:
Component: SPCCSS

Cause

Code inefficiencies with CORBA thread processing.

Resolution

This is resolved by Spectrum_10.03.00.PTF_10.3.027 for 10.3.0.  PTF_027 does require D04a to be installed.  This will also resolved inSpectrum_10.03.02.BMP_10.3.003 for 10.3.2 once released and will be included in 10.4 once released.  The release notes show:

Symptom : SpectroSERVER performance problems, causing OC to red-box but not failover to secondary.
        Resolution: Resolved several SpectroSERVER performance problems due to very high scale, and improved failover 
    mechanism to properly failover to secondary when the primary is in busy/hung state.
        (DE409335, 01321042)