Spectrum MySql Upgrade fail from 22.2.4 to 23.3.8

Products

• DX NetOps
• CA Spectrum

Issue/Introduction

When upgrading Spectrum from 22.2.4 to 23.3.8, the upgrade failed running the mysql.cus script.

********** Started: Mon Mar  7 18:40:51 2022 **********
running Custom Script Install-Tools/CUS/mysql.cus
 Error(s) occurred. Please see /opt/CA/Spectrum/Install-Tools/LOGS/21.2.8.0.32_20220307/mysql.log.

Script Install-Tools/CUS/mysql.cus has FAILED

The following is seen in the mysql.log:

******* Upgrading mysql *******

 stop_mysqld --login-path=myclient
Shutting down mysqld...
******* Starting mysql *******
 wait_for_mysql_to_actually_startup upgrade mysql --login-path=myclient
Waiting for mysql to start up... --login-path=myclient
Waiting for mysql to start up... try count 1 
.....
Waiting for mysql to start up... try count 50 
Final attempt to reach mysql... 
**^G Error during mysqladmin --defaults-file=/opt/CA/Spectrum/mysql/my-spectrum.cnf --login-path=myclient ping

This same error is seen in knowledge document "Spectrum SRM Upgrade failed due to mysql.cus" However, after following the resolution steps, the reinstall also failed.

Environment

Version: Any
Component: Upgrade

Resolution

Fortunately, in this instance, a snapshot was made of the system before the upgrade. The following directions were used to resolve the issue.

• Load the snapshot
• Log into the system as the user that owns the Spectrum installation
• Ensure mysql is running
• Run mysqldump to take a backup of the mysql databases:
  • cd to the $SPECROOT/mysql/bin directory and run the following command to backup the databases replacing <PASSWD> with the password for the root user:
  • cd to the $SPECROOT/mysql/bin directory and run the following command to backup the databases replacing <PASSWD> with the password for the root user:
• Move the databases.sql file to a safe place outside the $SPECROOT directory
• Ensure all of the Spectrum processes are shutdown except the SRAdmin process
• Remove the entire $SPECROOT/mysql directory
• Run the upgrade
• After the upgrade completes, move the databases.sql to the $SPECROOT/mysq/bin directory and run the following command to load the databases replacing <PASSWD> with the password for the root user:
• ./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD> < databases.sql

Additional Information

One thing to note and this is VERY important. The mysql databases, especially the reporting database on the SRM system, can be very large.

Ensure there is enough available space for this procedure. There should be at lease 2.5 times the size of the mysql databases available.

To find out how large the databases are, run the following command from the $SPECROOT/mysql/directory:

du ./data

The value of the ./data directory is what to look at. Make sure there is at least 2.5 times the available space.