Unbind MySQL service instance fails with error like "There was a problem when connecting to a MySQL server"

Products

• VMware Tanzu Application Service

Issue/Introduction

When attempting to unbind a MySQL service instance from an application on Apps Manager, it fails with error about connecting to a MySQL server as shown by image below. 

Command "cf unbind-service" would get same error message, which could also be found in MySQL service broker logs.

{"timestamp":"####.693226576","source":"on-demand-service-broker","message":"on-demand-service-broker.unbind.unknown-error","log_level":2,"data":{"binding-id":"537aaa1b-##-##-##-f7e4aea8230f","correlation-id":"9795437b-##-##-##-a64e98e334c4","error":"There was a problem when connecting to a MySQL server. Check database availability, persistent disk usage, and network availability, then try again.","instance-id":"7626f775-##-##-##-df90951c3855","requestIdentity":"ccc99e35-##-##-##-1b32095ba987","session":"11476"}}

 

Environment

MySQL service tile for TPCF

Cause

The MySQL service instance is a Leader-Follower HA cluster. However, the inspect errand of the corresponding BOSH deployment of the MySQL service instance shows that the mysql cluster is not in good leader-follower state. As shown in image below, one mysql instance is follower while the other is in "unknown" state. 

 

Resolution

Try to run make-leader errand to make the mysql instance in "unknown" state the leader role. The BOSH CLI command would be like,

$ bosh -d service-instance_7626f775-##-##-##-df90951c3855 run-errand make-leader --instance=mysql/d02cb916-##-##-##-a5879aa5ec12

After the errand completes successfully, the mysql cluster would be in leader-follower state. Run inspect errand to confirm.

$ bosh -d service-instance_7626f775-##-##-##-df90951c3855 run-errand inspect

Then attempt to unbind the MySQL service instance again, which should be successful.