SpectroSERVER MLS on RHL is at 100% CPU, pstacks show CsModelDomainSrvc::reg_attr_watch

book

Article ID: 197007

calendar_today

Updated On:

Products

CA Spectrum

Issue/Introduction

The SpectroSERVER MLS on RHL is at 100% CPU and is causing Spectrum to be unusable.  With the automatic performance minidumps that are created as well as manual pstacks taken we can see the following in the stacks.

 

Thread 1 (Thread 0x7fb0453ddcc0 (LWP 12861)):

#0  __strlen_sse2_pminub () at ../sysdeps/x86_64/multiarch/strlen-sse2-pminub.S:39

#1  0x00007fb039bf03e1 in CORBA_String_var::CORBA_String_var(CORBA_String_var const&) () from /opt/SPECTRUM/lib/libvport64_r.so

#2  0x00007fb03a180ba8 in VISIIOP::ProfileBody::toTaggedProfile() () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0

#3  0x00007fb03a19714f in VISDelegate::is_equivalent(CORBA_Object*, CORBA_Object*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0

#4  0x00007fb03a2a959a in CORBA_Object::_is_equivalent(CORBA_Object*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0

#5  0x00007fb044d043c2 in CsOrbMdlAttrWatcher::reg_callback(CsCAttrValWatchCB*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#6  0x00007fb044ce992d in CsModel::get_orb_watcher(CsCAttrValWatchCB*, CsModelHandle const&, unsigned int, CsWatchEnum::CsWatchType_e, void*, int, CsSecurityIf const*, int*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#7  0x00007fb044ce9ce6 in CsModel::reg_attr_watch(CsAttrWatchReq const*, CsCAttrValWatchCB*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#8  0x00007fb03fe9e9d2 in CsDModelIf::reg_attr_watch(CsAttrWatchReq const*, CsCAttrValWatchCB*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/libgserv.so.1

#9  0x00007fb044cfff17 in CsModelDomainSrvc::reg_attr_watch(CsModelHandle const&, CORBA_ULongSeq const&, CsCAttribute::CsCAttrReadMode_eList const&, CsCAttrValWatchCB*, CsSecurityIf const&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#10 0x00007fb044d002df in CsModelDomainSrvc::startWatchAttrValsOfModelsInt(CORBA_ULongSeq const&, CORBA_ULongSeq const&, CsCAttribute::CsCAttrReadMode_eList const&, CsCAttrValWatchCB*, CsSecurityIf const&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#11 0x00007fb044d0063f in CsModelDomainSrvc::startWatchAttrValsOfModelsByIDs(CORBA_ULongSeq const&, CORBA_ULongSeq const&, CsCAttribute::CsCAttrReadMode_eList const&, CsCAttrValWatchCB*, CsSecurityIf const&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#12 0x00007fb044c2d4fc in CModelDomainItcM::processItcRequest(ITC_Request_Parms*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1

#13 0x00007fb04228b5fb in ITC_Request_Parms_Corba::processRequest_TransferExceptions() () from /opt/SPECTRUM/lib/libitc.so.1

#14 0x00007fb04228a9b7 in ItcWorkQueue::process_work_item(ItcQdItem*) () from /opt/SPECTRUM/lib/libitc.so.1

#15 0x00007fb04228a802 in ItcWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libitc.so.1

#16 0x00007fb03fc20a70 in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1

#17 0x00007fb04495c211 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1

#18 0x00007fb0389e7190 in ?? () from /lib64/libc.so.6

#19 0x0000000000000000 in ?? ()

 

 

After a SpectroSERVER stop and restart the problem is no longer seen.  What can we do to prevent this from happening again in the future?

Cause

Currently is a known issue under "TA1436604 perf issue with CsModelDomainSrvc::reg_attr_watch"  that is scheduled to be fixed in a future release

Environment

Release : 10.4.1, 10.4.2.x

Component : Spectrum Core / SpectroSERVER

RHL

Resolution

As a workaround you can add the following to the $SPECROOT/SS/.vnmrc to avoid the schedules from looping.

watch_registration_schedule_batch_size=10