search cancel

Archive Manager corruption with ./ArchMgr -repair.

book

Article ID: 6815

calendar_today

Updated On:

Products

CA Spectrum DX NetOps

Issue/Introduction

We saw corruption on the events table of the Archive Manager as follows:

Error opening the DDM database. One or more tables are corrupt.

Recovery options:

  1. Run `ArchMgr -repair` to attempt recovery

  2. Load a valid DDM savefile using `ddm_load <savefile>`

  3. Initialize the DDM database using `ddm_load -i <LANDSCAPE_HANDLE>`

Segmentation fault (core dumped)

 

However the "ArchMgr -repair" did not resolve the issue and gave the following errors. 

 

ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:

SHOW COLUMNS FROM event LIKE 'node_id' - Table './ddmdb/event' is marked as crashed and last (automatic?) repair failed

Database corruption detected:

    ddmdb.event - Record at pos: xxxxxxxxx is not remove-marked

    ddmdb.event - record delete-link-chain corrupted

    ddmdb.event - Corrupt

Repairing...

    ddmdb.event - Can't create new tempfile: './ddmdb/event.TMD'

    ddmdb.event - Operation failed

May 08 09:13:53 : ArchMgr successfully repaired database.

 

We verified that there was enough free disk space to run the utility and tried to initialize the database but again got errors.

./ddm_load -i 0xxxxxxxx

SPC-SHD-29021:

ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:

SHOW COLUMNS FROM event LIKE 'node_id' - Table './ddmdb/event' is marked as crashed and last (automatic?) repair failed

 

We then tried unsuccessfully to use the mySQLcheck utility, to force a repair of the table but got the following errors.

 

./mysqlcheck --defaults-file=../my-spectrum.cnf -uroot -proot --repair --force ddmdb 

ddmdb.attribute OK 

ddmdb.db_descriptor OK 

ddmdb.event 

error : Can't create new tempfile: './ddmdb/event.TMD' 

status : Operation failed 

Cause

Corruption in the DDMDB database.

Environment

All MySQL databases that use MyISAM storage e.g. the DDMDB prior to Spectrum 10.3 or DDMDB in 10.3 and above which have not been converted to InnoDB.

This resolution does not apply to DDMDB that have been converted to InnoDB engine in 10.3 and above, and also not the SRM Reporting database which has been using InnoDB engine since Spectrum 9.3. For InnoDB engine database repair, use MySQLCheck - see reference below in "Additional Information"

Resolution

The resolution is to use the offline MySql repair utility myisamchk

1 - Log into the SpectroSERVER system as the user that owns the Spectrum installation

2. If on Windows, start a bash shell by running "bash -login"

3 - Stop the Archive Manager (if running)

    a - cd to the $SPECROOT/bin directory and enter the following command:

./cmdc localhost 2 ARCHMGR

    b. Ensure the Archive Manager is not running

4. Stop MySql

    a - cd to the $SPECROOT/bin directory and enter the following command:

./cmdc localhost 2 MYSQL

    b. Ensure the MySql is not running

5. cd to the $SPECROOT/mysql/bin directory and execute the following command:

./myisamchk -r -n -f --sort_buffer_size=2000M ../data/ddmdb/event.MYI 

5. Start MySql

    a - cd to the $SPECROOT/bin directory and enter the following command:

./launchinstdbapp localhost MYSQL y MYSQL.OUT

    b. Ensure the MySql is running

6. Start Archive Manager

    a - cd to the $SPECROOT/bin directory and enter the following command:

./launchinstdbapp localhost ARCHMGR y ARCHMGR.OUT

    b. Ensure the Archive Manager is running

Additional Information

If repairing an InnoDB engine database, use mysqlcheck. See MySQL manual for reference: https://dev.mysql.com/doc/refman/5.7/en/mysqlcheck.html