After Spectrum OneClick Integration with SpectroSERVER, the SpectroSERVER does not startup

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

After integrating VNA 3.74 with OneClick and modeling devices using SDN Manager, the Secondary SpectroSERVER will fail and no longer startup after Online backup.

If the Primary SpectroSERVER is stopped it will no longer startup and you must restore a SSDB before the VNA integration in order to start the SpectroSERVER.

During startup you will see lines in the VNM.OUT like this:

-----

...

Oct 10 22:06:22 : /opt/spectrum/SS/SpectroSERVER

     is now ready on port 0xbeef...

 

mprotect failed, errno=12

... (5883 times) ...

mprotect failed, errno=12

? mts: mts_mmap_block::Vfree_memory failed to unmap addr 0x7f858ea5b000 len 540672

-----

 

If you change the MTS executable to Non_MTS this will cause a proper crash that looks like

 

(gdb) where
#0  0x00007fc849d605ce in NewUnixContext::NewUnixContext(Thread*, unsigned int) () from /opt/SPECTRUM/lib/libmoot.so.1
#1  0x00007fc849d6133d in Thread::Thread(char const*, void (*)(...), unsigned int, unsigned char, states, void*) ()
   from /opt/SPECTRUM/lib/libmoot.so.1
#2  0x00007fc84822246f in IHExtendedFaultDomainManager::trig_model_activate(CsModelHandle const&, int) ()
   from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1
#3  0x00007fc84a05846f in CsIHActWatcher::notify(CsIncidentParm const*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#4  0x00007fc84a0c3d18 in CsModel::activate_model(int, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#5  0x00007fc84a0304f5 in CsActivateScheduler::process_queued_work(CsWorkNode const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#6  0x00007fc845311452 in CsWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libwkmgr.so.1
#7  0x00007fc845310b33 in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1
#8  0x00007fc849d60161 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1
#9  0x00007fc83e5a3180 in ?? () from /lib64/libc.so.6
#10 0x0000000000000000 in ?? ()

 

 

Environment

Release : 10.3

Component : Spectrum Core / SpectroSERVER

Cause

On startup the IH (inference handler) for Extended Fault Domain Manager utilizes to many resources that cause the SpectroSERVER to crash or error out (with MTS) when VNA is installed because many more fault domains are added.

Resolution

Please request Spectrum_10.03.02.PTF_10.3.238a from Support

This fix was also added to 10.4.1 and subsequent releases.

Additional Information

To enable No_MTS on the SpectroSERVER 

1. Stop the affected SpectroSERVER and wait for it to shutdown completely.

2. rename SpectroSERVER executable to SpectroSERVER_bak in $SPECROOT/SS

3. copy SpectroSERVER_noMTS from $SPECROOT/SS/support to $SPECROOT/SS

4. rename SpectroSERVER_noMTS to SpectroSERVER

5. Make sure SpectroSERVER has the same permissions as the previous SpectroSERVER_bak file.

6. Start the SpectroSERVER