The run-errand upgrade-all-service-instance on mysql service failed after we upgraded the stemcell version

Products

VMware Tanzu for MySQL

Issue/Introduction

After upgrading the stemcell version and applying changes to MySQL tile on OpsMan UI, the run-errand upgrade-all-service-instance on the mysql service instance failed with the following error:

$ bosh -d <mysql-service-instance-name> run-errand upgrade-all-service-instance

Task 8973130

Task 8973130 | 16:55:41 | Preparing deployment: Preparing deployment
...
Task 8973130 | 16:55:54 | Preparing deployment: Preparing deployment (00:00:13)
Task 8973130 | 16:55:54 | Preparing deployment: Rendering templates (00:00:03)
Task 8973130 | 16:55:57 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 8973130 | 16:55:58 | Updating instance mysql: mysql/xxxx-xxx-xxx-xxxx (0) (canary)
Task 8973130 | 16:55:58 | L executing pre-stop: mysql/xxx-xxx-xxx-xxx (0) (canary)
Task 8973130 | 16:55:59 | L executing drain: mysql/xxx-xxx-xxx-xxxx (0) (canary)
Task 8973130 | 16:56:45 | L stopping jobs: mysql/xxx-xxx-xxx-xxxxx (0) (canary)
Task 8973130 | 16:56:46 | L executing post-stop: mysql/xxxx-xxxx-xxxx (0) (canary)
Task 8973130 | 17:01:43 | L installing packages: mysql/xxxx-xxxx-xxxx (0) (canary)
Task 8973130 | 17:02:14 | L configuring jobs: mysql/xxxx-xxxxx-xxxxx (0) (canary)
Task 8973130 | 17:02:14 | L executing pre-start: mysql/xxxx-xxxxx-xxx-xxx (0) (canary) (00:07:09)
L Error: Action Failed get_task: Task xxxx-xxx-xxx-xxxx- result: 1 of 7 pre-start scripts failed. Failed Jobs: pxc-mysql. Successful Jobs: loggregator_agent, mysql-restore, bpm, bosh-dns, user_add, antivirus.
Task 8973130 | 17:03:07 | Error: Action Failed get_task: Task xxx-xxx-xxx-xxxxx result: 1 of 7 pre-start scripts failed. Failed Jobs: pxc-mysql. Successful Jobs: loggregator_agent, mysql-restore, bpm, bosh-dns, user_add, antivirus.

Task 8973130 Started Tue Dec 17 16:55:41 UTC 2024
Task 8973130 Finished Tue Dec 17 17:03:07 UTC 2024
Task 8973130 Duration 00:07:26
Task 8973130 error

Capturing task '8973130' output:
Expected task '8973130' to succeed but state is 'error'

Exit code 1

Environment

TAS

MySQL Tile

Cause

Checking the failing mysql service instance, we can see 3 of the mysql vms are still using the old stemcell 1.423, and only 1 of the mysql vm got updated to the new stemcell 1.555 that's causing the run-errand upgrade-all-service-instance to failed.

$ bosh -d <mysql-service-instance-guid> vms

Deployment 'service-instance_xxxx-xxxx-xxx'

Instance Process State AZ IPs VM CID VM Type Active Stemcell

mysql-jumpbox/xxx-xxx-xxxx running az-sldc 10.253.190.72 vm-3201da43-a6b2-48d0-ae2f-adb904fea0cb xlarge true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.423
mysql/xxxx-xxxx-xxxx-xxxx running az-sldc 10.253.190.71 vm-f7cb96f4-66ec-41bc-b801-3775684c44ac xlarge true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.423
mysql/xxxx-xxxx-xxxx-xxxx running az-rldc 10.253.190.69 vm-c5ac1fad-b1a3-483c-85d6-29da676d59c0 xlarge true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.423
mysql/xxxx-xxxx-xxxx-xxxx running az-rldc 10.253.190.70 vm-0edacfd2-5d9f-4cc1-ba55-a2cd5cd09958 xlarge true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.555

4 vms

Resolution

Note: Before attempting the resolution steps, we recommended that you backup the mysql service instance database in case you need to recover it.

Follow these steps to resolve the failing upgrade-all-service errand:

Try scaling the mysql service from 3 to 1 node to got it to run with 1 node cluster.
After getting the mysql service instance running with 1 node using the old stemcell version, then we run the upgrade-all-service-instance errand to get service instances updated to the new stemcell.
Now, we can scale up the mysql service instance from 1 node to back to 3 nodes cluster and updating to the new stemcell version.