SpectroSERVER is not Starting - Db::open: Invalid argument

Products

• DX NetOps
• CA Spectrum

Issue/Introduction

I've tried to restore a backup, but getting an error when starting the SpectroSERVER that I've never seen before: 

 

$ cd /opt/spectrum/bin
$ ./startSS.pl
Removing SpectroSERVER lock file
Started: SpectroSERVER Daemon, PID = 14144

Jan 30 14:24:56 : /opt/spectrum/SS/SpectroSERVER starting
Jan 30 14:24:56 : MTS enabled, thread heaps = 9
**************************************************************************

CA Technologies, A Broadcom Company
One CA Plaza
Islandia, NY  11749 USA
Copyright (c) 2020 CA Technologies, A Broadcom Company
All Rights Reserved

This software is protected by Copyright law and international treaties.

~
~

**************************************************************************
Version 10.4.2.002

Jan 30 14:24:56 : Validating and loading SPECTRUM Intelligence Modules.
Jan 30 14:24:58 : All SPECTRUM Intelligence Modules successfully loaded.

SPC-SHD-29021:
**************************************************************************
The following output is intended to facilitate debugging, please forward
to CA technical support.  The files listed are not part of the customer
installation of SPECTRUM.
**************************************************************************

Jan 30 14:24:59 ERROR TRACE at CsSSDbRp.cc(658): table open failed  @  TableImpl.cc:661
      (0x16)  Db::open: Invalid argument
Could not open the database. VNM exiting.
Landscape not initialized. VNM exiting.
Jan 30 14:24:59 : /opt/spectrum/SS/SpectroSERVER has successfully shut down.

Environment

Release : 20.2, 21.2, 22.2

Cause

The SpectroSERVER was not stopped gracefully and the SSdb had become corrupted.

Resolution

** Verify that a recent SSdb backup exists before proceeding **

In this instance, a restore of the SSdb database did not completely resolve the issue. It appears that one of the database table files .db/.ix 
   had become corrupt and was not being replaced by the restore.

 If the backup is compressed (.gz) uncompress the backup

cd $SPECROOT/SS-DB-Backup
gzip -d <backupfileName>.SSdb.gz

-  Check for a hung SpectroSERVER process   

ps -ef  | grep -i SpectroSERVER

     ** If found kill the process ( kill -9 <pid of SS>)

- Check for the .VNMDB.LOCK (remove if found)

cd $SPECROOT/SS/
rm -f .VNMDB.LOCK

- Remove the existing database files in case they are corrupt

cd $SPECROOT/SS/
rm -f *.db *.ix

verify that these have been removed

ls -la *.db *.ix

Run as the spectrum user for all below

- restore from the backup, this command must be run from the SS folder as the spectrum user

cd $SPECROOT/SS/
../SS-Tools/SSdbload -il ../SS-DB-Backup/<backupfileName>.SSdb

- If the restore completes successfully please try and start the SpectroSERVER as spectrum

cd $SPECROOT/bin/
./startSS.pl