Scheduler Errand 'register-broker' completed with error 404 Not Found

Products

• VMware Tanzu Application Service

Issue/Introduction

While upgrading Scheduler tile, the following error can be found in the errand stderr output and/or Apply Change logs:

Updating service broker scheduler-for-pcf as admin...
FAILED

Stderr     Job (abc-123) failed: The service broker rejected the request. Status Code: 404 Not Found. Please check that the URL points to a valid service broker.

1 errand(s)

Errand 'register-broker' completed with error (exit code 1)

Exit code 1

Cause

This issue can occur under certain conditions, including:

The user performs a jump upgrade of the MySQL tile (*example: from v3.1 to v3.3) while there is a concurrent upgrade of the Scheduler tile. This can cause the new MySQL (v3.3 in this example) to refuse to create additional bindings for outdated service instances until they get upgraded. In turn, this leads to the 'deploy-scheduler errand' to fail and leaves the Scheduler app/tile in a bad state. Since Scheduler was in bad state due to mysql issue, it can cause Apply Changes/errands to fail.

 

Notable:

*(1) In general the MySQL and Sceduler upgrade example mentioned above is okay, but if the MySQL tile's "upgrade-all-service-instances errand" is disabled, the MySQL instances are left at the old version (v3.1 using our example above).

*(2) Even if the MySQL service instances are upgraded after the above items, Scheduler may not be able to recover from its bad state automatically.

Resolution

To recover from this, the user can:

1. Delete all 3 Scheduler apps via Appsman UI (with "delete route" option unchecked).

2. Confirm the scheduler mysql service instance is upgraded. (cf upgrade-service scheduler-broker-mysql)

3. Ran Apply-Changes on Scheduler, with all options checked/selected

 

Notable:

The easiest and recommended way to avoid this issue is to run the mysql service upgrade errand before upgrading the scheduler tile.

Additional Information

It is encouraged to keep the 'upgrade all-service-instances on', or not running the scheduler upgrade, manually upgrading the scheduler service instances, then doing the scheduler upgrade.