MySQL Service Instance leader VM has the Role set as unknown.

Products

• VMware Tanzu Application Service

Issue/Introduction

The below command can be used to identify the events surrounding the service instance. 

Identify the Service Instance the guid using the cf CLI command below.
cf service SERVICE_INSTANCE --guid

SSH into the MySQL VM in the CF deployment.
bosh -d <cf-deployment> ssh mysql/12345

Replace the SERVICE_INSTANCE_GUID and execute the query to see events that occurred around the service instance. (to verify if there was a delete command initiated)
sudo mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf -e "select * from ccdb.events where actee = 'SERVICE_INSTANCE_GUID';"

Once you confirm that the service instance is not being deleted. (The word delete should not appear in the output)

We can execute the commands below to identify the state of the service instance.
bosh -d service-instance_MY_INSTANCE run-errand inspect

If the Role is set as unknown (Role: unknown) and Read Only is set to false (Read Only: false) for only one VM, we can determine that the Read Only: false is the leader VM.
We have to also verify that there are no missing data on the VMs. The GTIDs should be identical on the two VMs.

Resolution

Once we verify that the GTIDs are the same we can manually set the leader VM to resolve the issue.

Execute the below command to resolve the issue where the Service Instance was not deleted but an error is returned during the upgrade suggesting the service instance is orphaned.
bosh -d service-instance_MY_INSTANCE run-errand make-leader --instance=mysql/12345_the_leader_Instance

The Apply Change to upgrade the service instance should be successfully completed.