Upgrading Healthwatch fails with pxc throwing "Upgrade is not supported after a crash or shutdown" error

Products

• VMware

Issue/Introduction

Upgrading Healthwatch fails on grafana
 

Expected task '674923' to succeed but state is 'error'
                      L Error: 'grafana/f7e56023-d1d5-4a2b-a575-faf8f3072138 (0)' is not running after update. Review logs for failed jobs: grafana
Task 674923 | 02:42:51 | Error: 'grafana/f7e56023-d1d5-4a2b-a575-faf8f3072138 (0)' is not running after update. Review logs for failed jobs: grafana

There is actually 2 vms that are not in running state:


1) Grafana vm

check logs /var/vcap/sys/log/grafana/grafana.log show its unable to connect to pxc-mysql that is why its failing
 

logger=sqlstore t=2023-09-16T21:42:01.915431869Z level=info msg="Connecting to DB" dbtype=mysql
logger=migrator t=2023-09-16T21:42:01.922281772Z level=error msg="alert migration failure: could not get migration log" error="failed to check table existence: driver: bad connection"

2) pxc vm

checking logs  /var/vacp/sys/log/pxc-mysql/mysql.err.log and the error below is thrown
 

2023-09-16T17:15:55.016541Z 3 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 5.7.37-40, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html 2023-09-16T17:15:55.016640Z 3 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error. 2023-09-16T17:15:55.851982Z 3 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine 2023-09-16T17:15:55.852158Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed. 2023-09-16T17:15:55.852182Z 0 [ERROR] [MY-010119] [Server] Aborting

Environment

Product Version: 3.6

Resolution

The error on pxc-mysql is thrown because the pxc-mysql version transitions from 8.0.30 requires the MySQL database server to be gracefully shutdown before it can be upgrading. It is also rewriting the old ib_logfile* files into a new /var/vcap/store/pxc-mysql/#innodb_redo/* files, which it does not want to do if the previous instance was crashed.

What are ib_logfile*?
ib_logfile0 and ib_logfile1 are for, they are the InnoDB Redo Logs. This should never be erased or resized until a full normal shutdown of mysqld has taken place. If mysqld ever crashes, just start up mysqld. It will read across ib_logfile0 and ib_logfile1 to check for any data changes that were not posted to the the double write buffer in ibdata1. It will replay (redo) those changes. Once they are replayed and stored, mysqld becomes ready for new DB Connections. 

Here are steps to resolve this issue:

1. back up ib_logfile* by copying under /var/vcap/store/pxc-mysql directory into another directory i.e /tmp 
2. delete ib_logfile* under /var/vcap/store/pxc-mysql directory. Please note that deleting this files may occur partial data loss. 
3. check if pxc-mysql successfully restarted using monit summary