MySQL Upgrade All Service Instance Errand Fails due to AZ Error

Products

• VMware Tanzu Application Service

Issue/Introduction

Error seen in logs during running Upgrade All Service instances errand:

unexpected status code: 500. description: contact your operator, service configuration issue occurred.

 

and in MySQL Broker vm logs

Plans do not have compatible AZs. Please contact your operator.

 

Cause

MySQL broker runs a check to validate whether the AZ changes could result in losing persistent disks. Usually persistent disks are tied to a specific AZ.

The error, "Plans do not have compatible AZs. Please contact your operator.", comes from a MySQL broker check that validates whether the azs changes could result in losing persistent disks. Usually persistent disks are tied to a specific az and migration is either not supported or requires special steps. If previously a user configured azs: [az1, az2, az3] and creates some VMs, there could be vms in az "az1" and recreating that VM in "az2" would result in data loss.

Resolution

Here are steps to resolve the issue:

1. Make sure you have the original AZs. These are the AZs used when the service instances were created. Check AZ configuration for all MySQL plans and update it to original AZs:

MySQL Tile -> Settings -> Plans -> MySQL Availability Zones

And Apply Change with upgrade all service instances errand turned on. If the upgrade all service instances errand still fails, proceed to step 2.

 

2. If you have existing service instances

Check all the AZs configured by getting the manifest

bosh -d service_instance-GUID manifest > service_instance-GUID.yml

make a backup of the manifest before editing

cp service_instance-GUID.yml service_instance-GUID-backup.yml

Edit service_instance-GUID.yml by checking the AZs configured and update it to original AZs accordingly.

Here is a sample section of the manifest:

 vm_type: small
  stemcell: only-stemcell
  persistent_disk_type: "10240"
  azs:
  - az1
  - az2
  - az3
  networks:
  - name: ####

Then deploy manually using:

bosh  -d service_instance-GUID deploy service_instance-GUID.yml

Additional Information

Useful docs on Adding and Removing AZs can be found here:

https://bosh.io/docs/azs/#adding-and-removing-azs-to-a-deployment 

Please note on some of the problems that might be encountered:

• bosh does not migrate persistent disk contents across availability zones. Persistent disks attached to an instance (VM) that is moved to another AZ will be orphaned and eventually deleted by Bosh.
• Singleton instances will face downtime while being recreated. If it has persistent disks attached the data on the disk will not be migrated.
• Moving an instance (VM) with a statically assigned IP address will fail. Bosh will create the new instance before deleting the old instance. This means that at the creation time of the new instance, the static IP will still be attached to the old instance.
   

The MySQL broker attempts to guard against these problems by refusing the redeploy a service instance if any of its VMs might be orphaned.