Creating a new Scheduler fails with "Service broker error: Only one Scheduler instance can be created per space.” but no scheduler exists

Products

• Operations Manager

Issue/Introduction

When trying to create a Scheduler service instance in a space without any Scheduler instances existing in that space, you see the following error:

Service broker error: Only one Scheduler instance can be created per space.

The reason for this error is that the Scheduler broker database gets out of sync with the Cloud Controller (CC) database. As a result, CAPI thinks there is no Scheduler instance but the Scheduler broker rejects creating a new one because it thinks one already exists.

Resolution

To resolve the problem, you'll have to clean up the orphaned service instance from the Scheduler broker database. 

Note: Please be careful with this process because this will delete records from the database. It is strongly recommended that you take a backup of the database prior to deleting any records.

1. Get the space GUID where you are seeing the error with the following:

cf space <SPACE_NAME> --guid

2. Confirm there is no Scheduler instance in the current space with: 

cf services

3. Collect information from system org and scheduler space:

cf t -o system -s scheduler
cf service scheduler-broker-mysql --guid

4. Login to the above sheduler-broker-mysql database.

5. Find the deployment with MySQL service GUID:

bosh ds --column=name | grep <GUID>

6. SSH into the MySQL instance:

bosh -d <DEPLOYMENT> ssh mysql/0
sudo -i
mysql --defaults-file=/var/vcap/jobs/mysql/config/mylogin.cnf

7. Confirm the database and clean up the orphaned Scheduler instance: 

use service_instance_db;

8. Find the service instance GUID with:

select guid from service_instances where space_guid='<SPACE_GUID>';

9. Confirm that there are no binding apps with:

select * from service_bindings where service_instance_guid='<SERVICE_INSTANCE_GUID>';

10. Then you can delete the orphaned service instance with: 

delete from service_instances where space_guid='<SPACE_GUID>';