Spectrum 10.3 OLB not working

Products

Spectrum

Issue/Introduction

Since the update from Spectrum 10.2 to 10.3 (or to 10.3 + BMP_10.3.001) you may experience the following issues:

- At SpectroSERVER startup, you may notice high delays during the loading of models (eg. 15 minutes vs 3 minutes prior the upgrade)

- OLB backup may fail with the following debug messages in vnm.out file:

Nov 29 14:12:02 OLB TRACE at CsIHOB.cc(3873): Performing Backup
Nov 29 14:12:02 OLB TRACE at CsIHOB.cc(3878): Model Handle - 0xc6300008
Nov 29 14:12:02 OLB TRACE at CsIHOB.cc(1574): Initializing temporary database directory
Nov 29 14:12:02 OLB TRACE at CsIHOB.cc(883): One of the OLB attributes(BackupNow, NextBackupDateTime, AutomaticBackups) change has triggered OLB.
Nov 29 14:12:04 OLB TRACE at CsIHOB.cc(1688): Directory name - /backup/spectrum-10/SS-DB-Backup/db_20181129_1412.SSdb.tmpdir
Nov 29 14:12:04 OLB TRACE at CsIHOB.cc(1689): File name name - /backup/spectrum-10/SS-DB-Backup/db_20181129_1412.SSdb
Nov 29 14:12:04 OLB TRACE at CsIHOB.cc(1703): Done initializing temporary database directory - 0
Nov 29 14:12:04 OLB TRACE at CsIHOB.cc(1742): Copying the database
Nov 29 14:12:04 OLB TRACE at CsIHOB.cc(883): One of the OLB attributes(BackupNow, NextBackupDateTime, AutomaticBackups) change has triggered OLB.
Nov 29 14:12:07 OLB TRACE at CsIHOB.cc(1768): Pausing VNM operations
Nov 29 14:12:07 OLB TRACE at CsIHOB.cc(1788): Copying database files
Nov 29 14:12:08 OLB TRACE at CsIHOB.cc(1821): Continuing VNM operations
Nov 29 14:12:08 OLB TRACE at CsIHOB.cc(1847): VNM was paused for 0 minutes,4 seconds
Nov 29 14:12:08 OLB TRACE at CsIHOB.cc(1869): Compress value set for backup DBs : 1
Nov 29 14:12:08 OLB TRACE at CsIHOB.cc(883): One of the OLB attributes(BackupNow, NextBackupDateTime, AutomaticBackups) change has triggered OLB.
Nov 29 14:12:10 OLB TRACE at CsIHOB.cc(2077): Creating SSdb file
Nov 29 14:12:10 OLB TRACE at CsIHOB.cc(2091): Command to be executed : 0x7f69df2487b0
Nov 29 14:12:41 OLB TRACE at CsIHOB.cc(2099): start_process_return_ticket return value : 0x7f6a0705f0f8
Nov 29 14:12:41 OLB TRACE at CsIHOB.cc(2118): SSdbsave return bad exit code - -1
Nov 29 14:12:41 OLB TRACE at CsIHOB.cc(2139): Done reading SSdb file - 6
Nov 29 14:12:41 OLB TRACE at CsIHOB.cc(1898): Done copying the database - 6

and with a SSdbsave crash:

2018-12-10T14:56:31.650174+01:00 fluffy kernel: SSdbsave[31518]: segfault at d9cb000 ip 0000003107e89aa3 sp 00007ffeaaf18ea8 error 4 in libc-2.12.so[3107e00000+18a000]
2018-12-10T14:56:33.922488+01:00 fluffy abrt[31529]: Saved core dump of pid 31518 (/usr2/SPECTRUM/SS-Tools/SSdbsave) to /var/spool/abrt/ccpp-2018-12-10-14:56:31-31518 (215855104 bytes)
2018-12-10T14:56:33.927883+01:00 fluffy abrtd: Directory 'ccpp-2018-12-10-14:56:31-31518' creation detected
2018-12-10T14:56:34.026375+01:00 fluffy abrtd: Executable '/usr2/SPECTRUM/SS-Tools/SSdbsave' doesn't belong to any package and ProcessUnpackaged is set to 'no'
2018-12-10T14:56:34.026390+01:00 fluffy abrtd: 'post-create' on '/var/spool/abrt/ccpp-2018-12-10-14:56:31-31518' exited with 1
2018-12-10T14:56:34.026398+01:00 fluffy abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2018-12-10-14:56:31-31518'

Core was generated by `/usr2/SPECTRUM/SS-Tools/SSdbsave -cm /backup/spectrum-10/SS-DB-Backup/db_201812'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003107e89a5c in memcpy () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.209.el6_9.2.x86_64 libgcc-4.4.7-18.el6.x86_64 libstdc++-4.4.7-18.el6.x86_64
[?1034h(gdb) where
#0 0x0000003107e89a5c in memcpy () from /lib64/libc.so.6
#1 0x00007f6481e8216c in CsBuffer::pack(unsigned char*) const () from /opt/SPECTRUM/lib/libGlobl.so.1
#2 0x00007f64821a8a0b in attrpack(void const*, CsAttrDesc::CsAttrType_e, unsigned char*) ()
from /opt/SPECTRUM/lib/libVPapi.so.1
#3 0x00007f6482b47cc1 in CsDbModelRep::pack(unsigned char*) () from /opt/SPECTRUM/lib/libssdbl.so.1
#4 0x00007f6482b41d96 in CsDbLandscapeRep::pack(CsMemMCatalog*, CsOPackedFile&) ()
from /opt/SPECTRUM/lib/libssdbl.so.1
#5 0x0000000000406668 in SSdbsave::main(int, char const**) ()
#6 0x0000003107e1ed1d in __libc_start_main () from /lib64/libc.so.6
#7 0x0000000000404309 in _start ()
...

Environment

Spectrum 10.3 + BMP_10.3.001 on Linux platform.

Cause

Empty Trunking port type models causing vLan port list to grow high, causing OLB and SS failure.

Resolution

Apply 10.03.00.BMP_10.3.002 that includes the fix for the following issue:

(DE391733, 01228068)
Symptom: Empty Trunking port type models causing vLan port list to grow high, causing OLB and SS failure.
Resolution: Now vLan port list will not grow and will not lead to OLB and SS failures.

However, just installing the patch is not enough to clear the issue as you will need to delete and re-discover the vLAN devices.
To identify the problematic devices that need to be deleted and re-discovered you can do the following steps:

1) With  the 10.03.00.BMP_10.3.002 already installed, stop SpectroSERVER

2) modify the .vnmrc and add the line:
VLAN_DEBUG_ENABLED=TRUE

3) start SpectroSERVER

From the debug lines in the vnm.out search for "CsIIBList - port type list count". You may notice a number of models that have a very high number of  port type list count.
eg.
CsIHvLanConfig::trig_model_create for model 0xc63e3ff4
CsIIBList - port type list count 264857
...
CsIHvLanConfig::trig_model_create for model 0xc632cd71
CsIIBList - port type list count 216111
...
CsIHvLanConfig::trig_model_create for model 0xc6328e65
CsIIBList - port type list count 31174
...
CsIHvLanConfig::trig_model_create for model 0xc631e699
CsIIBList - port type list count 33919
...

4) Via Locater -> Models -> By Model Handle identify these models, probably they will be Container of Type vLan

5) Open the  Component Detail of these containers and in the Device View you can see the list of devices part of vLAN that need to be deleted and re-discovered.

6) Once you have re-discovered the problematic devices, remove the VLAN_DEBUG_ENABLED=TRUE from .vnmrc and stop/start SpectroSERVER. You will notice a significant improvement in performance during the load of models and OLB will start working again.

Additional Information

OnLine Backup debug can be enabled on the VNM model under the 'Dynamic Debugging' section