Primary SpectroSERVER crashes in a Fault Tolerant environment after Spectrum upgrade from 10.1.1 to 10.02.00
Article ID: 5083
Updated On:
Products
Issue/Introduction
After a successful upgrade of the primary SpectroSERVER to 10.02.00 in a Fault Tolerant environment, the primary SpectroSERVER crashes soon after starting.
The following is an example of what is seen in the stack trace of the dmp/core file
STACK_TEXT:
0000004c`f939f480 00007fff`bdd6e4ca : 0000004c`fd751410 0000004c`fe7b6ab0 0000004c`f1306b60 00000000`00012aca : libgas!AlarmSnoozeService::alarm_snooze_updated+0xce [d:\spectrum\churchill\cm\windows\10.02.00.00.244\gas.a\managers\src\alarmsnoozeservice.cc @ 488]
0000004c`f939f4e0 00007fff`bdd6d2ce : 00000000`00000000 0000004c`f1306b60 0000004c`f12abc78 00007fff`c7115fa2 : libgas!CsLocalAlarmTable::notify_clients_of_change+0x25a [d:\spectrum\churchill\cm\windows\10.02.00.00.244\gas.a\managers\src\cslalarmt.cc @ 666]
0000004c`f939f5d0 00007fff`bdd7f1bd : 0000004c`fb61eb00 0000004c`fb61eb00 0000004c`fd3bedb0 0000004c`f1438e60 : libgas!CsLocalAlarmTable::change+0x2e2 [d:\spectrum\churchill\cm\windows\10.02.00.00.244\gas.a\managers\src\cslalarmt.cc @ 458]
0000004c`f939f620 00007fff`bdd751f9 : 0000004c`fb61eb00 0000004c`fb61eb00 0000004c`f1438e60 00007fff`bdb7e40e : libgas!CsModelAlarmService::write_attributes+0x3d [d:\spectrum\churchill\cm\windows\10.02.00.00.244\gas.a\managers\src\csmdlalserv.cc @ 5667]
0000004c`f939f650 00007fff`bdd6f27b : 0000004c`2ee00000 00000000`00000000 00000000`00000001 0000004c`fb61eb00 : libgas!CsModelAlarmService::gen_secondary_alm_events+0x2ad [d:\spectrum\churchill\cm\windows\10.02.00.00.244\gas.a\managers\src\csmdlalserv.cc @ 9032]
0000004c`f939f6e0 00007fff`bdf6434d : 00000000`00000011 00010417`2ee0008d 00007fff`00000001 00000000`00000011 : libgas!CsMASIf::gen_secondary_alm_events+0x93 [d:\spectrum\churchill\cm\windows\10.02.00.00.244\gas.a\managers\src\csmasif.cc @ 1884]
0000004c`f939f710 00007fff`bdfe7e69 : 0000004c`f12e4438 0000004c`f12e4438 0000004c`f12adf78 0000004c`f12adf78 : libsskrnl!CsLandscape::CsLandscape+0x871 [d:\spectrum\churchill\cm\windows\10.02.00.00.244\sscor.a\lscpe\src\cslscpe1.cc @ 692]
0000004c`f939f9d0 00007fff`c72f769b : 00000000`00000000 0000004c`f1425140 00000000`00000000 00000000`00000000 : libsskrnl!VNMServer::vnm_start_up+0x1a5 [d:\spectrum\churchill\cm\windows\10.02.00.00.244\sscor.a\src\vnmserver.cc @ 774]
0000004c`f939fa90 00007fff`d2cd79bf : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : libmoot!call_func+0x47
[d:\spectrum\churchill\cm\windows\10.02.00.00.244\moot.a\nt\src\ntcontext.cc @ 83]
0000004c`f939fac0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseFiberStart+0x2f
Environment
Cause
The Primary SpectroSERVER has a new snooze alarm functionality that is conflicting with the secondary SpectroSERVER that does not yet have this new functionality.
Resolution
The Spectrum 10.02.00.PTF_10.2.003 patch is available to prevent the crash from occurring. The following is from the Spectrum 10.02.00.PTF_10.2.003 patch release notes:
Symptom: Spectrum 10.2 crashed with AlarmSnoozeService in the stack trace.
Resolution: The SpectoSERVER will no longer crash.
Please contact CA Spectrum Support for a copy of the Spectrum 10.02.00.PTF_10.2.003 patch.
This issue has been fixed in CA Spectrum 10.2.1 release:
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/spectrum/10-2/release-information/issues-resolved.html
Symptom: Spectrum 10.2 crashed with AlarmSnoozeService.
Resolution: The SpectoSERVER will no longer crash with AlarmSnoozeService.
(10.2.1, DE265177, 00633417)
Additional Information
Recover from possible database corruption.
If the patch is installed after the SpectroSERVER has crashed, you will need to recover from possible database corruption prior to starting the SpectroSERVER process. During the installation of Spectrum 10.02.00, there will be three new database files created in the $SPECROOT/SS directory:
legacy.SSdb - This database file is a catalog only database containing a clean catalog from the installation. A new legacy.SSdb file is created every time a Spectrum installation is run. The version of the legacy.SSdb file will be the version of Spectrum installed at that time.
There will be a file date/time stamped with the word "after" in the file name. For example, "db_20170111,1.after.SSdb". This is commonly referred to as the ".after" database file. This is also a catalog only database containing a clean catalog from the installation. The version of the .after file will be the version of Spectrum installed at that time.
There will be a file date/time stamped that will contain the models and catalog of the previous version of Spectrum installed. For example, "db_20170111,1.SSdb". If installing Spectrum 10.02.00 over the top of Spectrum 10.01.01, this database will be version 10.01.00,
Please do the following to recover from possible database corruption after the crash and after installing the PTF_10.02.003 patch:
1. Log into the SpectroSERVER system as the user that owns the Spectrum installation
2. On Windows, open a bash shell by running "bash -login"
3. cd to the $SPECROOT/SS directory
4. Remove the $SPECROOT/SS/.VNMDB.LOCK file
5. Run the following command to initialize the database to the legacy catalog:
../SS-Tools/SSdbload -il legacy.SSdb
6. Run the following command to load the models only from the models and catalog save from before the upgrade was done:
../SS-Tools/SSdbload -m <models and catalog database save from the previous version of Spectrum>
Example: ../SS-Tools/SSdbload -m db_20170111,1.SSdb
7. Start the SpectroSERVER as normal.
It is highly recommended to make a new full database save that will now be of a database having 10.02.00 and PTF_10.02.003 installed.
Feedback
