Spectrum SS hangs after upgrade to 24.3.2 and higher on Windows
Article ID: 388854
Updated On:
Products
Issue/Introduction
After upgrading Spectrum (NetOps Fault) to 24.3.2 and higher on Windows systems configured to use Spectrum's Legacy Landscape Handle only, we have seen:
1. Running a SSdb Online Backup (OLB) hangs the SS.
2. Shutting down or stopping the SS freezes and does not complete.
In both situations, waiting does not resolve the issue. The SS needs to be killed and the SSdb needs to be restored from the legacy.SSdb which contains a snapshot of the db from directly after upgrade.
The following errors are seen in the SS/VNM.OUT when immediately starting the SS or might not be seen until after running the OLB causing stack dump creation (SpectroSERVER.dmp and .moot.trace updates):
Feb 19 15:01:28 WARNING at ItcQdItem.cc(381): Request 46333 accepted, but not complete after 600085ms, cs delta=774547
Feb 19 15:01:45 WARNING at ItcQdItem.cc(381): Request 49925 accepted, but not complete after 360048ms, cs delta=752986
Feb 19 15:02:32 ERROR TRACE at CsIHCrMdlEv.cc(350): Model Name is not set after re-evaluation for mh:0x47aa50
Feb 19 15:02:45 WARNING at ItcQdItem.cc(381): Request 49925 accepted, but not complete after 420050ms, cs delta=1724219
Feb 19 15:02:47 WARNING at ItcQdItem.cc(374): Request 60738 accepted, but not complete after 60003ms, cs delta=1849825. Triggering moot trace.
Feb 19 15:03:45 WARNING at ItcQdItem.cc(381): Request 49925 accepted, but not complete after 480053ms, cs delta=8347379
Feb 19 15:03:47 WARNING at ItcQdItem.cc(381): Request 60738 accepted, but not complete after 120005ms, cs delta=8502949
Feb 19 15:04:45 WARNING at ItcQdItem.cc(381): Request 49925 accepted, but not complete after 540061ms, cs delta=10844745
Feb 19 15:04:47 WARNING at ItcQdItem.cc(381): Request 60738 accepted, but not complete after 180012ms, cs delta=10902772
Feb 19 15:05:45 WARNING at ItcQdItem.cc(381): Request 49925 accepted, but not complete after 600075ms, cs delta=10832385
Environment
24.3.2 and higher on Windows only
Cause
In 24.3.2+, the Spectrum Berkeley database was upgraded to version Berkeley DB 18.1.40 which causes a situation with Spectrum caching for databases running Legacy Landscape Handle
https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/24-3/release-information/Features-and-Enhancements-2432.html
Starting with this release, the Spectrum SpectroSERVER database now uses Berkeley DB 18.1.40. The SpectroSERVER database is an in-memory persistent Berkeley DB to store models, alarms, associations, and model types.
Resolution
A resolution will be added to the code in NetOps 24.3.7.
For those on or upgrading to 24.3.2 through 24.3.6 please verify if the SSdb is using legacy landscape handle.
This can be done by reviewing the $SPECROOT/.installrc SSLHANDLE entry. For example:
SSLHANDLE: 4 <------ Legacy Landscape Handle
SSLHANDLE: 64 <------ Huge Landscape Handle
Once verified, reducing the SSdb cache size resolves this issue. Add the following to the .vnmrc:
ssdb_cache_size_kbytes=10000
Restart the SS (the CLI action code to reread the vnmrc does not work for this setting).
Please note:
- The configuration change should be done before the upgrade on Windows for any of the "24.3.2 to 24.3.6" versions.
- If any user upgrades without the config change, they still can make the change before starting the SpectroSERVER.
- For any system that are already upgraded and the SpectroSERVER is running without the configuration change, they can still add the configuration change and restart the SpectroSERVER.
- For any system that are already upgraded and the SpectroSERVER hang behavior is observed, the SpectroSERVER process will need to be killed. Make the configuration change and restart the SpectroServer by loading the last known good DB. If directly after upgrade, this will be legacy.SSdb which is created at the end of upgrade.
Feedback
