Spectrum crashed and try to start SpectroSERVER shows error "database was not gracefully closed"
search cancel

Spectrum crashed and try to start SpectroSERVER shows error "database was not gracefully closed"

book

Article ID: 113595

calendar_today

Updated On:

Products

CA Spectrum DX NetOps

Issue/Introduction

We recently experienced a SpectroSERVER crash in CA Spectrum. We tried to start the database back up but now we are seeing an error in the Spectrum Control Panel: 

"ERROR: This application has been started using a Spectrum database which was not gracefully closed. 
The application will now shut down to allow you to load and initialize a previously-saved good database."

How do we get our server back up and running? 

Environment

Release: All Supported Releases

Component: SPCCSS - SpectroSERVER Core

Resolution


This feature is meant to prevent corruption-related issues. The appropriate course of action is to restore to a last known good backup from before the time of the crash. There are multiple ways to accomplish this: 

NOTE: Before running ANY of these steps, be sure to log into the system as the user that owns the Spectrum installation

1. Restore from Spectrum Control Panel

This can be done by opening the Spectrum Control panel and selecting the Restore Database button. When it asks you to initialize the database, select yes, then select a db save file from before the crash. Once complete, start the SpectroSERVER.

2. Restore from a command line

cd to <SPECROOT>/SS

  • cd to <SPECROOT>/SS-DB-Backup/
  • run: gunzip <your_backup_filename>   (to extract the database file)
  • cd to <SPECROOT>/SS/    (you must be in the <SPECROOT>/SS/ directory to perform the restore operation)
  • run: ../SS-Tools/SSdbload -il ../SS-DB-Backup/<your_backup_filename>
  • Once complete, start the SpectroSERVER from Spectrum Control Panel
  • or start the Server from a command line: cd to <SPECROOT>/bin and run ./StartSS.pl


NOTE: SSdbload and SSdbsave utility are in the SS-Tools directory but MUST be run from /SS directory to work properly. See the Additional Information section below for more info on SSdbload and SSdbsave commands. 

NOTE: When the server starts, it creates a lock file for the user that started the database. It resides in <SPECROOT>/SS as .VNMDB.LOCK. Normally when the server is shut down gracefully, the LOCK file is removed. If the server gets shut down inadvertently (a crash or the cable got pulled, etc) the LOCK file remains and needs to be deleted manually, then the load of the database or other db maintenance can proceed. 


If the above steps fail to load a good backup, there may be issue with a corrupt Database Catalog. If this happens, you may need to load legacy.SSdb first, to first load an initial (empty) database, and specify Catalog only. Then load models on top from a backup SSdb. 

3. Restore legacy database (initial database) and load models on top

  • cd to <SPECROOT>/SS
  • run  ../SS-Tools/SSdbload -c legacy.SSdb
  • run ../SS-Tools/SSdbload -m ../SS-DB-Backup/<your_backup_filename>
  • Once complete, start the SpectroSERVER


NOTE: Starting in Spectrum 10.2, the legacy.SSdb is a copy of the database as it was prior to any upgrades. Previous to 10.2, the legacy.SSdb is a completely empty database. Starting from 10.2, legacy.SSdb is a database which contains Catalog and Models from the previous Spectrum version (if it was upgraded to the current version, as opposed to a new install). This is to allow a database save to exist immediately after an upgrade completes, as a failsafe should the upgrade fail or some other issue occur before the server can be started and backup after the upgrade. Thus, if loading a 10.2 and higher legacy.SSdb with ../SS-Tools/SSdbload -il legacy.SSdb, it will not appear "empty" but instead will contain models as they existed at the time of the upgrade. Loading legacy.SSdb Catalog Only will appear "empty". 

4. In some extreme cases is may be necessary to completely remove the database files prior to loading a backup.

NOTE: before getting this far, do try the above options (1-3) first. 

  • The SSdb files exist in the $SPECROOT/SS directory as ss*.db (data files) , ss*.ix (index file) , and *.log (log files)
  • Manually delete the files then restore the database using one of the above steps (1-3) 


5. If all else fails, please open a Support Ticket for assistance. Be sure to include the SpectroSERVER Logs for review: 

  • cd to <SPECROOT>
  • run ./bin/support/getSpectrumInfo.sh lite
  • upload <SPECROOT>/logs_<hostname>_<timestamp>.tar.gz to the case



NOTE: If you have also experienced a crash on Archive Manager DDM database, please see "Spectrum Archive Manager not starting after system crash."

Additional Information

Please reference the "SSdbload" and "SSdbsave" sections of the documentation for more information.

Powered by Wolken Software