DX NetOps Spectrum Fault Tolerant Archive Manager is Crashing

Products

• Network Observability

Issue/Introduction

Since upgrading DX NetOps Spectrum to a 25.4.x release, we have noticed that our Fault Tolerant Archive Manager has crashed. When running gdb on the core file, we are seeing the following stack:

#0 0x00007f7c4a415a74 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007f7c4c1de2df in VISMutex::lock() () from /opt/SPECTRUM/lib/libvport64_r.so
#2 0x00007f7c4be2897d in vbsec::CSIV2ServerReqInt::receive_request_service_contexts(PortableInterceptor::ServerRequestInfo*) () from /opt/SPECTRUM/lib/libvbsec64.so
#3 0x00007f7c4c79f11d in VISPISrvrReqIntAdapter::preinvoke(VISServerRequestInfo&, VISClosure&) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#4 0x00007f7c4c79f797 in VISChainServerInterceptor::preinvoke(VISServerRequestInfo&, VISClosure&) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#5 0x00007f7c4c7799c5 in VISServerReplyHandler::preinvoke_intc(VISPOA*, VISActivationRecord*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#6 0x00007f7c4c740f54 in VISPOA::pre_invoke(VISInvokingInfo*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#7 0x00007f7c4c74eee2 in VISPOA::preinvoke(ProtocolEngine::ServerEngine*, VISObjectId*, VISInvokingInfo*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#8 0x00007f7c4c7d9581 in VISPOAProtocolConnector::invoke(ProtocolEngine::RequestMessage*, unsigned long, unsigned long) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#9 0x00007f7c4c797305 in VISChainClientInterceptor::invoke(ProtocolEngine::RequestMessage*, unsigned long, unsigned long) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#10 0x00007f7c4c78c94f in VISDelegate::invoke(CORBA_Object*, CORBA_MarshalOutBuffer*, CORBA_ExceptionList*) () from /opt/SPECTRUM/lib/liborbcore64_r.so.8.0
#11 0x00007f7c508c1ed1 in CsCEventDomain::stopWatchEvents(CsCEventWatchCB*) () from /opt/SPECTRUM/lib/libssorb.so.1
#12 0x00000000004d9599 in CsEventDomainConnInfoWork::cleanup() ()
#13 0x00007f7c50fe8017 in WatchQueueListener::performWork(void*) [clone .cold.33] () from /opt/SPECTRUM/lib/libgserv.so.1
#14 0x00007f7c50d81ff9 in ItcNativeQueue::processWork() () from /opt/SPECTRUM/lib/libitc.so.1
#15 0x00007f7c50d82159 in ItcNativeQueue::runCallback(void*) () from /opt/SPECTRUM/lib/libitc.so.1
#16 0x00007f7c4a4131ca in start_thread () from /lib64/libpthread.so.0
#17 0x00007f7c4a06e953 in clone () from /lib64/libc.so.6

Cause

When the CORBA service shuts down, and the ArchMgr is trying to clean-up the stopWatchEvents watches which invokes a loopback CORBA call that locks an already-destroyed mutex which crashes the ArchMgr.

Resolution

A code fix has been added to prevent this crash in the 25.4.10 release.