Mysql tile Upgrade All Service Instances errand timing
Article ID: 418398
Updated On:
Products
Issue/Introduction
The upgrade all service instances errand is run by default when you upgrade the tile. This errand should also be used when there are changes that need to be implemented on the service instances.
Environment
Tanzu for MySQL on Tanzu Platform
Cause
The errand may take a long time to complete if there is a large number of service instances in the platform.
Upgrade all service instances errand has a polling interval of 60 seconds (as per config in /var/vcap/jobs/upgrade-all-service-instances/config/config.yml `polling_interval: 60`) and the mysql upgrade requires two bosh tasks. The first bosh task is "create deployment" and the second task is to "run errand configure-tz".
The way errand works is:
1. Broker launches the upgrade on an instance. i.e starting first task (create deployment).
2. The first task completes in less than a minute.
3. Because of polling interval of 60 seconds, broker will not check this instance until it hits the polling interval again, at which point it detects the create deployment task is completed and launches a second required bosh task for the service instance (configure tz).
4. Task "configure tz" completes.
5. Because of a polling interval timer, broker will not detect it until it hits the interval time and checks the instance again. At this point it will mark it as upgraded and completed.
Resolution
The poll interval is not configurable. The other bottle neck is Bosh Director has a limited number of worker threads (configurable) that will slow down the speed of upgrades as well. You can speed up the the service instance upgrade by doing more instances simultaneously by adjusting the number of service instances that are upgraded at any one time. Also make sure the bosh director worker count to be 1 higher than the max instance upgrades.
BOSH Tile -> Director Config --> Director Workers
Mysql tile -> Service Instance Upgrades --> Number of Simultaneous Upgrades
Feedback
