MySQL V2 Upgrade Errand Fails when Reinstalling the Tile

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

When deployment MySQLv2 Tile the following error is observed

[upgrade-all-service-instances] 2018/02/13 00:34:26.414249 Upgrade failed for service instance df64a9a6-fc60-40fb-8362-e1177299fc7a: 
unexpected status code: 500. description: expected status 200, was 401. 
Response Body: {"code":600000,"description":"Require one of the scopes: bosh.admin, bosh.e378d55a-7f7d-40af-9cdb-115161315da3.admin, 
bosh.teams.pivotal-mysql-26556b51b9b864556c3b.admin, bosh.read, bosh.e378d55a-7f7d-40af-9cdb-115161315da3.read"}

 

Environment

Cause

A user will see this error in the following scenario:

• Operator Installs MySQLv2 tile successfully
• Users create new On Demand Service instances
• The operator turns off the "Delete All Service Instances and de-register Broker" errand
• The operator then deletes the MySQLv2 tile from Operations Manager
• The operator re-installs the MySQLv2 tile and the deployment succeeds
• Errand "Upgrade all service instances" will then fail with the status code 500 error

 

Resolution

Follow these steps to get the environment back into a state where the MySQLv2 tile can install successfully.  Other tiles such as Pivotal Cloud Cache, Redis, and RabbitMQ which offer On Demand services will have similar symptoms. These steps can also be used to clean up the cloud controller and Bosh director. 

Note- This procedure assumes users have not created new On-Demand Instances after re-installing the tile. Also in this procedure, we assume the latest version of cf CLI and Bosh CLI version 2.

1. Purge all the existing On-Demand Service Instances from Cloud Controller Database

   cf purge-service-instance SERVICE_INSTANCE
   

2. Get a list of all the deployments and identify the MySQLv2 deployments. Replace "MY_BOSH_ENV" with your Bosh environment alias

   bosh -e MY_BOSH_ENV deployments
   

3. Delete all the On Demand Service Instance Deployments. In the below example replace "MY_BOSH_ENV" with your Bosh environment alias and "MY_DEPLOYMENT_NAME" with the name of MySQLv2 On Demand Deployment name.

   bosh -e MY_BOSH_ENV -d MY_DEPLOYMENT_NAME delete-deployment