Orphaned MySQL service instances block upgrade-all-service-instances errand from completing

Products

• VMware Tanzu Platform Data
• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu for MySQL

Issue/Introduction

Customer upgraded stemcell in their Tanzu for MySQL tile and applied changes, including the Upgrade All Service Instances errand. To their surprise the VMs for the service instances were not updated to the stemcell version specified in the tile.

Environment

Deployment 'pivotal-mysql-<guid>'
Instance                                                     Process State  AZ     IPs           VM CID                                   VM Type  Active  Stemcell
dedicated-mysql-broker/<guid>  running        AZ-##  ###.###.###.###  vm-<guid>  medium   true    bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.999

Deployment 'service-instance_<guid>'
Instance                                    Process State  AZ     IPs           VM CID                                   VM Type     Active  Stemcell
mysql/<guid>  running        AZ-##  ###.###.###.###  vm-<guid>  small.disk  true    bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.866

Deployment 'service-instance_<guid>'
Instance                                    Process State  AZ     IPs           VM CID                                   VM Type     Active  Stemcell
mysql/<guid>  running        AZ-##  ###.###.###.###  vm-<guid>  small.disk  true    bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.866

Deployment 'service-instance_<guid>'
Instance                                    Process State  AZ     IPs            VM CID                                   VM Type     Active  Stemcell
mysql/<guid>  running        AZ-##  ###.###.###.###  vm-<guid>  small.disk  true    bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.866

Cause

Out of 135 service instances (SI), 2 were orphaned (lacked a corresponding bosh deployment). These SI's blocked the errand from completing. Evidence appeared in the bosh task log for the errand; each iteration was noted as "Starting to process service instance 1 of 135" instead of counting each one and iterating to the next number.

Resolution

Collect and examine the bosh task log for the upgrade-all-service-instances errand:

Get the task number from Ops Manager, or from the bosh command line:

bosh tasks --recent=100 | grep errand

bosh task <task-number-for-errand> --debug > debug-task-log.txt

If you see evidence of orphaned SIs:

[upgrade-all-service-instances] 2026/01/13 01:24:47.020450 2 errors occurred:\n\n* failed to trigger operation for instance \"<guid>\": unexpected response status 502 when upgrading service instance \"<guid>\"; response body \"{\\\"description\\\":\\\"Service broker error: bosh deployment ''service-instance_<guid>'' not found

...collect the GUID's and purge the orphaned SI's following the procedure in this article. Once the orphaned SI's are gone, the upgrade-all-service-instances errand should be able to complete, and the VMs will be upgraded to the configured stemcell.