Cannot unbind MySQL service from app due to "Could not find an available leader"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

There are several situations that may result in this message. In this case, no monit jobs exist on the MySQL VM for a service instance's deployment during pre-start for the MySQL job. When checking /var/vcap/sys/log/mysql/mysql.err.log, it is seen that MySQL fails to start due to a missing bin log (i.e., mysql-bin.000039 for example).

mysqld: File './mysql-bin.000039' not found (Errcode: 2 - No such file or directory)
2020-06-30T18:57:59.899362Z 0 [ERROR] Failed to open log (file './mysql-bin.000039', errno 2)
2020-06-30T18:57:59.899368Z 0 [ERROR] Could not open log file
2020-06-30T18:57:59.899373Z 0 [ERROR] Can't init tc log
2020-06-30T18:57:59.899377Z 0 [ERROR] Aborting

This kind of issue is sometimes seen after a MySQL VM's persistent disk has filled up, possibly due to an attempt to resolve the full disk.

Environment

Product Version: 2.8

Resolution

To address this, SSH into the MySQL VM, and check the bin logs in the /var/vcap/store/mysql/data directory. The corresponding mysql-bin.index in the same directory should name the existing bin logs. If it doesn't, the mysql-bin.index file can be edited to match the existing bin logs. After adjusting the mysql-bin.index file to match the existing bin logs, if any, and recreating the MySQL VM for that service instance deployment, the service instance should be restored with all expected existing monit jobs.