Slowness of the model activation - class IHPeerLinkVerify

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

Model activation takes around 1 hour

Environment: Spectrum 10.4.1 in Windows

Integration with UIM via SNMP Gateway probe (not using spectrumgtw probe)

 

Two types of messages occur multiple times in VNM.OUT file:

Nov 11 19:38:29 WARNING at CsWorkSched.cc(184): Low thread resources detected. Work scheduler requesting 1min/50max threads, but not available.

Model 0x49cd6d "xxxxxxx"/UIMVirtualMachineHost - class IHPeerLinkVerify - 1407 seconds

Model 0x4146dd "UIM Unreported Inventory"/UnreportedUIMEntitiesContainer - class IHPeerLinkVerify - 1313 seconds

Model 0x4553f4 "yyyyyyyy"/UIMVirtualMachineHost - class IHPeerLinkVerify - 1393 seconds

Model 0x401452 "UIM Inventory"/UIMInventoryContainer - class IHPeerLinkVerify - 1411 seconds

Environment

Release : 10.4.1

Component : Spectrum Integrations

Cause

Through the Process Explorer, get the Stack from the SpectroSERVER process.

 

ntoskrnl.exe!KeSynchronizeExecution+0x2106
ntoskrnl.exe!KeWaitForMultipleObjects+0x135e
ntoskrnl.exe!KeWaitForMultipleObjects+0xdd9
ntoskrnl.exe!KeWaitForMutexObject+0x373
ntoskrnl.exe!KeStallWhileFrozen+0x1feb
ntoskrnl.exe!KeIsAttachedProcess+0x229
ntoskrnl.exe!KeSynchronizeExecution+0x4552
KERNELBASE.dll!WriteStateAtomValue+0x424
KERNELBASE.dll!WriteStateAtomValue+0x343
DBGHELP.DLL!SymCompareInlineTrace+0x754
DBGHELP.DLL!SymRegisterCallbackW64+0x10e
DBGHELP.DLL!SymGetModuleInfo+0x63
DBGHELP.DLL!ImagehlpApiVersionEx+0x80d7
DBGHELP.DLL!optdbgdumpaddr+0x267
DBGHELP.DLL!optdbgdumpaddr+0x1e5
DBGHELP.DLL!SymFunctionTableAccess64AccessRoutines+0x12a
DBGHELP.DLL!SymGetFileLineOffsets64+0x34dc
DBGHELP.DLL!MiniDumpReadDumpStream+0x16d17
DBGHELP.DLL!MiniDumpReadDumpStream+0x1860f
DBGHELP.DLL!MiniDumpReadDumpStream+0x17fde
DBGHELP.DLL!MiniDumpReadDumpStream+0x11d16
DBGHELP.DLL!StackWalkEx+0x4ce
DBGHELP.DLL!StackWalk+0x132
libGlobl.dll!CsSymbolInfo::print_stack+0xdd
libmoot.dll!IOEvent::setup_fd+0x1473
libmoot.dll!Thread::dump+0x13c
libmoot.dll!CsMootPerf::dec_thread_count+0x39c
libmoot.dll!CsMootPerf::dec_thread_count+0xd6
libwkmgr.dll!CsWorkScheduler::CsWorkScheduler+0x2c2
libwkmgr.dll!CsWorkQueue::CsWorkQueue+0x3e
libsskrnl.dll!CsMultiReqScheduler::CsMultiReqScheduler+0x46
libsskrnl.dll!CsMultiReqScheduler::get_scheduler+0xb7
libsskrnl.dll!CsLandscape::multi_model_request+0x71
libIHapi.dll!CsVnmLscpeHandle::multi_model_request+0x18
libmdlsvint.dll!ExtFaultCommon::sendMultiModelAction+0xe8
libmdlsvint.dll!IHExtendedFaultDomainContactManager::areAllModelsDown+0x18e
libmdlsvint.dll!IHExtendedFaultDomainContactManager::triggerModelContactLost+0x1cd
libmdlsvint.dll!IHExtendedFaultDomainContactManager::firstContactTriggerModel+0x57
libmdlsvint.dll!IHExtendedFaultDomainContactManager::startWatchOnTriggerModel+0x1b9
libmdlsvint.dll!IHExtendedFaultDomainContactManager::correlatesRelationChanged+0x383
libmdlsvint.dll!IHExtendedFaultDomainContactManager::virtualModelAdded+0x47
libnmsf.dll!IHNimsoftESXiHostDomainContactStatus::startMonitoring+0x496
libnmsf.dll!IHNimsoftESXiHostDomainContactStatus::trig_rel_change+0x16d
libsskrnl.dll!CsIHRelWatcher::notify+0x7cb
libsskrnl.dll!CsRelation::trig_rel_change+0x20e
libsskrnl.dll!CsRelation::notify_models+0x1fa
libsskrnl.dll!CsRelation::activate+0x90
libwkmgr.dll!CsWorkQueue::process_work_node+0xf
libwkmgr.dll!CsWorkScheduler::do_work+0x40
libmoot.dll!IOEvent::setup_fd+0x13e7
KERNEL32.DLL!CreateFiberEx+0x27f

 

ESX <-> PhysicalHost_NimsoftVmwareRedirector should be equal in number, it is a one-to-one mapping.

The root cause is due to 173,390 PhysicalHost_NimsoftVmwareRedirector models in the SSdb database.

Resolution

Download the attached CleanupNimsoftProxyRedirectorModels.sh script file from this KB article.

Copy it to the $SPECROOT/vnmsh/ directory.

Logged as Spectrum Install Owner account run the script. It may take 24 hours or more to finish, depending on the number of PhysicalHost_NimsoftVmwareRedirector models.

Before running the script, disable the scheduled OnLineBackup. But run an on-demand OnLineBackup.

bash -login

$ cd vnmsh

$ ./connect

$ ./CleanupNimsoftProxyRedirectorModels.sh

Once it is finished successfully, make an OnLineBackup of the SSdb database.