Scheduler Tile Upgrade Fails on deploy-scheduler Errand Due to MySQL Broker Unbind Timeout
search cancel

Scheduler Tile Upgrade Fails on deploy-scheduler Errand Due to MySQL Broker Unbind Timeout

book

Article ID: 450537

calendar_today

Updated On:

Products

VMware Tanzu for MySQL VMware Tanzu Platform VMware Tanzu Platform - Cloud Foundry VMware Tanzu Platform Core

Issue/Introduction

During an upgrade/deployment of the Scheduler for Tanzu Application Service tile, the deploy-scheduler errand fails during the application rollover phase.

The errand output displays an error indicating that it cannot unbind the p.mysql service instance (scheduler-broker-mysql), which prevents the old application from being deleted and blocking the rename step.

Errand Error Message:

failed: Service broker failed to delete service binding for instance scheduler-broker-mysql: Service broker error: There was a problem completing your request.  service: p.mysql, service-instance-guid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, broker-request-id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, operation: u
scheduler-broker-new scheduler-broker
-1
name 'scheduler-broker' already exists.
]]

 

Cause

This can happen if the backend MySQL service instance is not responding or if the VM is hung.

You might see the following error message on the MySQL broker VM under /var/vcap/sys/log/broker logs

Couldn't get read only information about URL https://<MYSQL_IP>:8443, error: GET https://<MYSQL_IP>:8443/status failed: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The deploy-scheduler errand uses a blue-green deployment strategy that requires cleanly unbinding and deleting the old application. When the errand requests the unbind, the Cloud Foundry Cloud Controller contacts the MySQL service broker. The broker then attempts to make an HTTPS GET request to the MySQL service instance VM (typically over port 8443) to retrieve its status and process the unbind.

While the underlying network and TCP routing are open and functioning, the service agent process running on the MySQL VM becomes hung or unresponsive. Because the agent fails to respond to the HTTP request, the broker's connection times out (context deadline exceeded). To prevent a state mismatch, the broker safely aborts the unbind request, which ultimately causes the Scheduler upgrade errand to fail.

Resolution

To resolve this issue, you must restart the hung processes on the affected MySQL service instance VM.

  1. Identify the affected IP: Review the MySQL service broker logs and locate the context deadline exceeded error to identify the specific IP address of the unresponsive MySQL node.

  2. Access the VM: Use the BOSH CLI to SSH into the unresponsive MySQL service instance VM: bosh -d <mysql-deployment-name> ssh <instance-group>/<instance-id>

  3. Restart Monit Services: Escalate to root privileges and restart all Monit processes to cleanly unfreeze the agent and database services:

    • Run: monit stop all

    • Wait for the processes to fully stop, then run: monit start all

  4. Verify: You can optionally test the unbind manually via the CF CLI (cf unbind-service scheduler-broker scheduler-broker-mysql) to confirm the broker can now communicate with the VM.

  5. Resume Upgrade: Return to the Ops Manager UI and click Apply Changes to re-run the deploy-scheduler errand and complete the upgrade.

Powered by Wolken Software