SpectroSERVER crash while evaluating Brocade device interfaces
search cancel

SpectroSERVER crash while evaluating Brocade device interfaces

book

Article ID: 273689

calendar_today

Updated On:

Products

DX NetOps CA Spectrum

Issue/Introduction

Spectrum 21.2.x and 22.2.x crashes on Brocade device configurations.

Windows Stack:

STACK_TEXT:  
000000df`56bde820 00007ffc`f657fefd : 00000000`00000000 00000274`610076f4 00000274`215ea140 00007ffc`0024844b : libbroc1000ss+0x210c
000000df`56bde970 00007ffc`f6589067 : 00000000`00000000 000000df`56bdedc0 00000000`00000000 00000274`00000001 : libmdlsvint!CsIHIfConfig::finish_if_change+0x839
000000df`56bdecc0 00007ffc`f65754d4 : 00000274`2e99c690 00000000`00000000 00000274`2e99c690 00000000`00000001 : libmdlsvint!CsIHIfConfig::interface_change+0x8f7
000000df`56bdefa0 00007ffc`f6b0c7b6 : 00000000`00000001 00000274`610076c8 00000274`3f99bc78 00000000`00000001 : libmdlsvint!CsIHIfConfig::trig_watch_change+0x2474
000000df`56bdf310 00007ffc`f6b73537 : 00000274`50c09af8 00000000`000100c3 00000274`556389b0 00000274`2cbe7a30 : libsskrnl!CsIHMdlAttrWatcher::notify+0x586
000000df`56bdf640 00007ffc`f6b7960c : 00000274`00000003 000000df`56bdf758 00000274`50a2d260 00000274`420e7880 : libsskrnl!CsModel::react_to_change+0x153
000000df`56bdf730 00007ffc`f6b8f6bf : 00000274`61007e70 00000000`00000000 00000000`00000000 00000000`00000000 : libsskrnl!CsModel::process_changes+0x4c
000000df`56bdf790 00007ffd`1add4efb : 00000274`0024844b 00000274`61007e70 00000000`00000000 00000274`745357f0 : libsskrnl!CsNotificationMgr::process_queued_work+0x63
000000df`56bdf7c0 00007ffd`1add45d4 : 00000274`40227950 00000000`00000000 00000000`00000000 00000274`74545950 : libwkmgr!CsWorkQueue::process_work_node+0xf
000000df`56bdf7f0 00007ffd`1b57783f : 00000274`40227950 00000000`00000000 00000000`00000000 00000000`00000000 : libwkmgr!CsWorkScheduler::do_work+0x40
000000df`56bdf820 00007ffd`2e2c0221 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : libmoot!IOEvent::setup_fd+0x13e7
000000df`56bdf850 00007ffd`319f172a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21
000000df`56bdf880 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserFiberStart+0x1a

RHEL:

(gdb) where
#0  BrocIHIfConfig::readSwFcPortTable (this=0x7f342633e220, mh=..., ifTable=0x7f33d3fdbde8) at BrocIHIfConfig.cc:148
#1  0x00007f347b0e11c1 in CsIHIfConfig::finish_if_change (this=0x7f342633e220, mh=...) at CsIHIfCnfg1.cc:1367
#2  0x00007f347b0f6f02 in CsIHIfConfig::interface_change (this=0x7f342633e220, mh=..., event_code=CsIHIfConfig::IF_COUNT_CHANGE, first_time=0, force_ip_addr_read=<optimized out>)
   at CsIHIfCnfg1.cc:1208
#3  0x00007f347b0d946c in CsIHIfConfig::trig_watch_change (this=0x7f342633e220, mh=..., other_mh=..., change=<optimized out>) at CsIHIfCnfg.cc:1757
#4  0x00007f347d2ed961 in CsIHMdlAttrWatcher::notify (this=0x7f33d33e4038, info=0x7f33355fde80, user_security=<optimized out>) at CsIHMAWatch.cc:229
#5  0x00007f347d364cae in CsModel::react_to_change (this=this@entry=0x7f340c0ece10, changes=...) at CsModel2.cc:875
#6  0x00007f347d36ede2 in CsModel::process_changes (this=this@entry=0x7f340c0ece10) at CsModel4.cc:631
#7  0x00007f347d38a9a9 in CsNotificationMgr::process_queued_work (this=0x7f346c40f800, work_node=<optimized out>) at CsNotifMgr.cc:342
#8  0x00007f347828f512 in CsWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libwkmgr.so.1
#9  0x00007f347828ea70 in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1
#10 0x00007f347cfe2211 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1
#11 0x00007f346daf2190 in ?? () from /lib64/libc.so.6
#12 0x0000000000000000 in ?? ()

 

RHEL /var/log/messages

Jan 22 19:21:35 <HOSTNAMEE> kernel: SpectroSERVER[2079677]: segfault at 20 ip <HEX> sp <HEX> error 4 in libbroc1000ss.so.1[7f055dbd8000+7000]

Environment

Release: 22.2.x

Resolution

  • This crash is resolved in Spectrum 23.3.2 which is now GA.
  • If the crash is still occurring in Spectrum 23.3.4, please open a support ticket and request the Spectrum_23.03.04.D259 debug patch.

Additional Information

If you are running 22.2.9 ONLY you can obtain a patch for this crash:

  • Spectrum_22.02.09.D224

 

Workaround to prevent the crash until the patch is applied or Spectrum is upgraded:

1.  Stop SS

2.  Stop processd

3.  Move (not rename) libbroc1000ss.dll OR  libbroc1000ss.so.1 from SPECROOT\SS\modules to somewhere on the box, not within SPECROOT

4.  Start processd

5.  Start SS

 

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/23-3/release-information/issues-resolved.html

Symptom:  Brocade devices return incorrect counts for swFCPortIndex and swFCPortType from the SW-MIB, causing the SpectroSERVER to crash. 
Resolution: With this fix, Spectrum adds a solution to ensure the validity of both swFCPortIndex and swFCPortType, which prevents the SpectroSERVER crash. (DE573809, 33507159, 23.3.2)

 

 

 

Powered by Wolken Software