Unable to Delete or Create RabbitMQ Service Instances

Products

• VMware RabbitMQ

Issue/Introduction

Symptoms:

This article provides a troubleshooting guide for when you are unable to delete or create RabbitMQ Service instances.

When attempting to delete or create a RabbitMQ service instance, it fails with the error message:

cf delete-service rabbitmq-service-instance
Really delete the service rabbitmq-service-instance?> yes
Deleting servicerabbitmq-service-instance in org ORG / space SPACE as USER..

FAILED
Server error, status code: 502, error code: 10001, message: Service instance order-api-rabbitmqold: The service broker rejected the request to https://pivotal-rabbitmq-broker.domain/v2/service_instances/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX. Status Code: 404 Not Found, Body: 404 Not Found: Requested route ('pivotal-rabbitmq-broker.domain') does not exist.

Environment

Cause

Following are the few reasons for why the RabbitMQ service broker is unable to process the request:

• RabbitMQ service broker encountered an error during serving the request.
• RabbitMQ service broker is not up and running.
• Rabbit Virtual Machines (VMs) are in an unknown state.

Resolution

Following are the troubleshooting steps you can take to ensure your RabbitMQ service broker is up and running:

1. Try running cf purge-service-instance rabbitmq-service-instance to see if it successfully deletes the instance. If not, proceed to the next steps.
2. Check the health of the VMs by using the command bosh cck. If Bosh CCK did a repair on the Rabbit Tile VMs successfully, try to create/delete RabbitMQ service and see if that resolves the issue. If Bosh CCK reported a problem but is unable to repair it, please contact Pivotal Support.
3. If Bosh CCK did not report any problems, go to Ops Manager console -> RabbitMQ Tile -> Status Tab. Check if all the VMs have been assigned to the specific IP. If the IPs show "unknown" under the IP column, create the VMs using  bosh recreate <job> <index>. Try to create or delete the service and see if this resolves the issue. 
4. Check the jobs with RabbitMQ service broker. Do Monit summary and see if rabbitMQ-broker and broker-route-register are running:

• If it is not running, try to run the processes by using monit start <job>. If the job starts successfully, try to create or delete a Rabbit service and see if the issue is resolved.
• If any of the processes are unable to start successfully, contact Pivotal Support.
• If the job(s) are already running, check the RabbitMQ service broker logs for the errors and contact Pivotal Support.