NetOps Portal MySql database fails to start

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

Cannot get to the PC web UI. Login to take a look and find all PC services down. Try to start mysql and it fails.

/opt/CA/MySql/data/hostname.err file has following errors;

WARNING [innoDB] Unable to open './netqosportal/temp_inclusion_items.idb'
Note Ignoring './netqosportal/temp_inclusion_items.idb' invalid tablespace in header
Note The log sequence number xxxxx in the system tablespace does not match the log sequence number in the ib_logfiles!
[ERROR] [innoDB] Plugin initialization aborted with error Generic error
[ERROR] [Server] Failed to initialize DD storage engine
[ERROR] [Server] Data Dictionary initialization failed
[ERROR] [Server] Aborting

Prior to startup issues I see a lot of WARNINGS:

Failed to preallocate data for file './netqosportal/tmp....idb', desired size xxxx operating system error number 22 - invalid argument. check that disk is not full or disk quota exceeded. Make sure the filesystem supports this function

The "ERROR" messages at startup are as follows (these were in sequence).

[ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 39867
[ERROR] [MY-012964] [InnoDB] Use –innodbn-directories to find the tablespace files. If that fails then use –innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s)
[ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error
[ERROR] [MY-010334] [Server] Failed to initialize DD storage engine
[ERROR] [MY-010020] [Server] Data Dictionary initialization failed
[ERROR] [MY-010119] [Server] Aborting

No other messages related to tablespace ID 39867 are found.

Environment

All supported DX NetOps Performance Management releases

Cause

Anti-Virus software running against the MySql data and files.

Resolution

Exclude all PC Portal MySql related files, directories, services are other components from Anti-Virus scans.

If problems continue shut down any active Anti-Virus software or agents running on the PC Portal host.

To recover the broken database follow these steps.

Ensure a valid backup is available.
1. If one is not available create a tar file backup of the (default path) /opt/CA/MySql/data directory.
2. Ensure all sub-directories and their contents are backed up.
Edit the /etc/my.cnf file.
1. Add the following line to the [mysqld] section.
  - innodb_force_recovery=1
Save the file changes and start the mysql service.
After mysql starts successfully edit the /etc/my.cnf file.
1. Remove the newly added innodb_force_recovery line.
2. Save the changes to the file.
3. Restart MySQL once more.

Additional Information

To ensure successful recovery in the face of catastrophic failures or unrecoverable corrupted databases ensure Portal backups are being generated on a regular scheduled basis.

See the Backup NetOps Portal documentation topic for additional information.