DX NetOps Performance Management 21.2.8 MySql Upgrade Tips
search cancel

DX NetOps Performance Management 21.2.8 MySql Upgrade Tips

book

Article ID: 235365

calendar_today

Updated On:

Products

DX NetOps CA Performance Management - Usage and Administration

Issue/Introduction

With the new r21.2.8 release of DX NetOps Performance Management comes an upgrade to the MySql DB utilized with the Portal web server.

The current 5.7 MySql release is upgraded to MySql 8.0.27.

One of the many benefits this provides is support for warm backups taken while PC services remain running.

There are some known things to review prior to executing the upgrade. These are MySql items that should be determined, and resolved where needed, prior to attempting the upgrade.

Failure to take these precautions can lead to a failed upgrade. In this case there are scenarios where a failure of the upgrade can lead to unrecoverable MySql databases.

Environment

All supported DX NetOps Performance Management PC Portal web server releases r21.2.7 and earlier.

Cause

Upgrade of MySql database version utilized with the DX NetOps Performance Management PC Portal web server host.

Resolution

IMPORTANT:

###########################

It is critical that you ensure valid MySql DB backups are taken before attempting the r21.2.8 upgrade. Failure to do so, in the face of upgrade errors or failures during the pre-install period for the MySql update, will likely result in corrupted and unrecoverable MySql databases. If this transpires the only resolution is recovery utilizing a reinstall of the previous release in use followed by restorating of the latest MySql database backups.

###########################

 

To backup the MySql databases follow the steps in the Back Up NetOps Portal Database section of the Back Up NetOps Portal documentation. It is always recommended to perform a full backup of the PC Portal system before starting an upgrade.

 

Steps to prepare for running the upgrade to 21.2.8?

  • If any schemas other than the netqosportal, em and other defaults are found to exist the installer will exit.
    • The installer will not allow a complete run until custom or non-default schemas are removed.
    • To determine what schemas are present run the following on the PC Portal host.
      • Open a terminal to the PC Portal host.
      • Run (default path shown) the following command, entering the password when prompted.
        • /opt/CA/MySql/bin/mysql -uroot -p -e "show databases;"
      • Sample results are seen here from a 21.2.x lab.
        • These are the only schemas that can be present before the upgrade will move forward.
        • [root@PC_Portal ~]# /opt/CA/MySql/bin/mysql -uroot -p -e "show databases;"
Enter password:
+--------------------+
| Database           |
+--------------------+
| information_schema |
| em                 |
| mysql              |
| netqosportal       |
| performance_schema |
| sys                |
+--------------------+
[root@PC_Portal ~]#
      • Have you installed Virtual Network Assurance (VNA) by pointing it to a 'shared externalized mysql instance'?
        • Some have done this in labs in older releases.'
        • Most often the PC Portal MySql instance would have been used.
        • If this is your situation please contact Support via a new support case for additional guidance before running the upgrade.
  • Is your MySql database externalized from PC Portal and installed on a different host?
    • In an environment with an external PC Portal MySql database you MUST upgrade the database on the external host BEFORE proceeding with the PC Portal upgrade.
  • Are you running r3.7.14 or older releases?
    • Be aware that 3.7 is reaching End of Service on March 31st, 2022.
    • These systems are required to upgrade to 3.7.15+ to get to MySql 5.7.
    • Then it can be updated to 21.2.8 and MySql 8.0
  • Are you running PC Portal on older RedHat 6.x release?
    • As of r21.2.8 we no longer provide support for RedHat 6.x releases.
    • This partly due to the MySql 8.0 upgrade and it's requirements.
  • Expect and plan for an estimated 25% increase in disk utilization.
    • This is due to the conversion of remaining MyISAM MySql tables to newer InnoDB table types.
    • Post upgrade we're using all InnoDB tables. No more MyISAM tables will be used.
    • The newer MySql 8.0 no longer supports the older MyISAM tables we still used in some parts of the Portal databases.
  • Running OpenSUSE Linux for PC Portal? Be aware of the new 'libuser' package requirement for OpenSUSE installs.
  • What if we've created new tables under a supported schema, or added custom schemas?
    • Results will depend on the table added, and whether or not it conforms to the newer MySql 8.0 requirements.
    • Engineering has suggested that we may be able to pick up the required changes as long as it is in the product schemas. Results are not guaranteed or supported.
    • If there is a stored procedure that's been added, one that isn't ours, we would not be able to handle it. Any problems resulting from it would not be supported.

 

What happens during the install run?

  • During the upgrade run for PC Portal users will be prompted to accept and acknowledge the successful generation of PC Portal MySql database saves prior to the upgrade to MySql 8.x in Portal release 21.2.8.
  • During the upgrade there are many internal schema changes required to migrate from the current MySql 5.7 version in use to the new MySql 8.0 version. These are ONLY performed for the default databases present in the product. It is often in these schema changes that things can go awry resulting in unrecoverable databases. It is vital to have successful backups before running this upgrade.
  • Engineering testing shows no appreciable time increase for the upgrade cycle while updating the tables from MyISAM to InnoDB.

 

What are possible failure points or causes?

  • Should there be errors during the MySql portions of the upgrade, how can we determine the severity? This is difficult to answer without specific details or errors. In general:
    • Failures during the pre-upgrade cycle are most often those that result in unrecoverable problems.
      • In most instances errors in this area would result in the need to:
        • Re-install the prior PM release that used the MySql 5.7 version.
        • Load the previously set aside database backups before trying the upgrade again.
    • Failures during the portions of the upgrade after the MySql pre-upgrade section are often recoverable.
      • These errors are often recoverable by re-running npc_shell commands at the direction of engineering, the same commands that the installer failed to run successfully.
      • If this is seen open a support case for further investigation.
      • Be sure to include a diagnostics logging package from the problem PC Portal server via the re.sh script.
      • Instructions for the re.sh script are here in the Unable to Resolve Issue documentation page.

 

Key log files for use in reviewing installation progress and results.

  • These are some of the key log files related to the MySql portion of this upgrade. Review these post upgrade as needed. Share them with support for problems. Default paths shown.
    • /opt/CA/MySql/mysql_initialize_results.txt
    • /opt/CA/MySql/mysql_upgrade_results.txt
    • /opt/CA/PerformanceCenter/InstallLogs/pre_mysql_upgrade.log.<Date/Time>
    • /opt/CA/PerformanceCenter/InstallLogs/post_mysql_upgrade.log.<Date/Time>

 

Known issues, troubleshooting and problem solutions.

  • Upgrading in an all IPv6 environment will result in failure post upgrade.
    • To resolve this we'll need to edit the bind-address value.
      • This is only defined in MySql 8.0 /etc/my.cnf files.
      • MySql 5.7 versions do not define this entry in their /etc/my.cnf file.
    • To resolve this take these steps:
      • Edit the /etc/my.cnf file. Find the bind_address entry.
        • Change it from using 0.0.0.0 to just simply *.
        • Should look like this after editing:
          • bind-address = *
        • Save the file changes.
      • This will allow for IPv6 connections to the database.
    • Restart the mysql service after making this change to implement it.
  • How to fix corrupted MySql post update to 8.x?
    • InnoDB is much more stable.
    • Generally if it can't fix it by itself it's often broken beyond repair.
    • The mysqlcheck command will still work. Need to use the -o argument.
  • If there are more than the expected default users, or those default users don't all have matching passwords, Portal login failures will be seen post upgrade.
    • Run the following to determine if you'll see this before running the upgrade.
      • Open a terminal to the PC Portal host.
      • Run (default path shown) the following command, entering the password when prompted.
        •  /opt/CA/MySql/bin/mysql -uroot -p -e "select user,host,authentication_string from mysql.user where user in ('root','netqos');"
      • Do we see two root and two netqos users, where all four have matching authentication_string values?
    • Sample output from a PC Portal 21.2.x lab.
      • These are the only root and netqos users that should be present.
      • They should all use the same password represented by the authentication_string value. All strings should match.
      • [root@PC_Portal~]# /opt/CA/MySql/bin/mysql -uroot -p -e "select user,host,authentication_string from mysql.user where user in ('root','netqos');"
Enter password:
+--------+-----------+-------------------------------------------+
| user   | host      | authentication_string                     |
+--------+-----------+-------------------------------------------+
| root   | localhost | *DE0F14483GAD1E4103A51757BD2E8A1FBF918454 |
| netqos | localhost | *DE0F14483GAD1E4103A51757BD2E8A1FBF918454 |
| netqos | %         | *DE0F14483GAD1E4103A51757BD2E8A1FBF918454 |
| root   | %         | *DE0F14483GAD1E4103A51757BD2E8A1FBF918454 |
+--------+-----------+-------------------------------------------+
[root@PC_Portal ~]#
    • If your system has mixed authentication_string values for those users, post upgrade a 403 error will be seen when trying to login to PC Portal.
      • Do you see a 403 error trying to login to PC Portal after upgrading to 21.2.8 and the authentication_strings don't match for all users?
        • Open a support case for assistance.
        • The steps to resolve this can, if not followed properly, result in more severe problems.
        • Support will assist in resolving this while avoiding more severe results.
  • If there are more users in the database than are present by default, the installer will fail to run.
  • Special characters in the MySql database password interfere with required checks for unexpected additional schemas in the MySQL database.
Powered by Wolken Software