The Archive Manager does not start SPC-OCP-10067 error

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

The Spectrum  Archive Manager fails to start with SPC-OCP-10067 error in a Fault Tolerant environment. 

The steps outlined in the following KB article did not fix the issue:

https://knowledge.broadcom.com/external/article?articleId=57274

 

$ ./ArchMgr.exe -repair

Dec 24 16:57:18 : ArchMgr started as user 'xxxxxxxxx'
Dec 24 16:57:18 : ArchMgr attempting to repair corrupted database

Database corruption detected:
        ddm.event - Partition EVENT_18_11_2023 returned error
        ddm.event - Corrupt
Repairing...
        ddm.event - OK
Dec 24 16:58:40 : ArchMgr successfully repaired database.

But the Archive Manager does not start. Rerun the repair.

$ ./ArchMgr.exe -repair

Dec 24 17:03:02 : ArchMgr started as user 'xxxxxxxxx'
Dec 24 17:03:02 : ArchMgr attempting to repair corrupted database

Database corruption detected:
        ddm.event - Partition EVENT_18_11_2023 returned error
        ddm.event - Corrupt
Repairing...
        ddm.event - OK
Dec 24 17:04:24 : ArchMgr successfully repaired database.

Environment

Spectrum: Any release

Cause

The DDM database is corrupted.

Resolution

1. Back up the DDM database on the good node (logged as Spectrum Install Owner account in a bash shell).

cd $SPECROOT/mysql/bin/
./mysqldump.exe --databases -uroot -p<password> ddmdb > backup_ddmdb.sql       (Windows)

./mysqldump --defaults-file=../my-spectrum.cnf --databases -uroot -p<password> ddmdb > backup_ddmdb.sql       (Linux)

2. Transfer the file over to the problematic node, in the $SPECROOT/mysql/bin/ directory.

3. On the problematic node, drop the DDM database.

cd $SPECROOT/mysql/bin/

./mysql.exe -uroot -p<password> ddmdb       (Windows)

./mysql  --defaults-file=../my-spectrum.cnf  -uroot -p<password> ddmdb        (Linux)

mysql> drop database ddmdb;

If it fails, try increasing the innodb_force_recovery parameter in the $SPECROOT/mysql/my-spectrum.cnf file. The default value is 0.

https://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html

Here are the steps:

Edit the $SPECROOT/mysql/my-spectrum.cnf file and add/update the following parameter:

innodb_force_recovery = 1       (example)

Stop/start the Spectrum MySQL server service to put the change in place.

To stop MySQL service service:

cd $SPECROOT/bin/

./cmdC.exe <hostname> 2 MYSQL
OR

./cmdC.exe localhost 2 MYSQL

 

To start MySQL service service:

cd $SPECROOT/bin/

./launchinstdbapp.exe <hostname> MYSQL y MYSQL.OUT
OR

./launchinstdbapp.exe localhost MYSQL y MYSQL.OUT

 

Ensure the $SPECROOT/mysq/data/ddmdb/ directory was removed during the drop operation.

Once you successfully dropped the DDM database, set the innodb_force_recovery back to 0 and stop/start the Spectrum MySQL service to put the change in place.

4. On the problematic node, recreate the DDM database.

cd $SPECROOT/mysql/bin/

./mysql.exe -uroot -p<password>        (Windows)

./mysql  --defaults-file=../my-spectrum.cnf  -uroot -p<password>         (Linux)

mysql> create database ddmdb;

 

5. On the problematic node, initialize the DDM database:

 

cd $SPECROOT/SS/DDM/

./ddm_load.pl -i -l <LH> 

 

6. On the problematic node, restore the DDM database (taken from the good node):

 

cd $SPECROOT/mysql/bin/
./mysql.exe -uroot -p<password> ddmdb < backup_ddmdb.sql      (Windows)
./mysql  --defaults-file=../my-spectrum.cnf  -uroot -p<password> ddmdb < backup_ddmdb.sql         (Linux)

 

7. Once the restore has finished successfully, start the Archive Manager.

Note: It will take some time to validate the restored DDM database.

https://knowledge.broadcom.com/external/article?articleId=32832