Spring Cloud Services (SCS) broker stops responding due to RabbitMQ being down
Article ID: 297136
Updated On:
Products
Support Only for Spring
Issue/Introduction
Creating a Spring Cloud Service (SCS) instance fails with the following error in the spring-cloud-broker app in the org system, space p-spring-cloud-services:
In SCS 1.x - SCS 2.x, RabbitMQ is being used as message bus between the SCS broker and worker. In this case, the broker sent out a service bind message to worker through RabbitMQ, but no response is received. The error indicates a problem with the RabbitMQ or worker.
2020-03-20T02:58:45.24-0700 [APP/PROC/WEB/0] OUT 2020-03-20 09:58:45.241 ERROR [spring-cloud-service-broker,4e769e7053741147,4e769e7053741147,false] 42 --- [nio-8080-exec-5] o.s.c.s.c.ServiceBrokerExceptionHandler : Unknown exception handled: 2020-03-20T02:58:45.24-0700 [APP/PROC/WEB/0] OUT org.springframework.messaging.MessagingException: No response received from 'spring.cloud.broker.5.bind' queue (likely due to timeout).
In SCS 1.x - SCS 2.x, RabbitMQ is being used as message bus between the SCS broker and worker. In this case, the broker sent out a service bind message to worker through RabbitMQ, but no response is received. The error indicates a problem with the RabbitMQ or worker.
Resolution
Check the status of spring-cloud-broker-worker app. If it's not working, fixing it is out of the scope of this article.
If it's working and you confirm the SCS worker didn't receive the message by checking worker logs, the problem is with the RabbitMQ cluster.
Open the RabbitMQ Management UI as explained here, and check the status of the cluster. It's highly probable that connections are being blocked due to a disk or memory alarm being triggered. Identify the cause of the resource being overused and release it.
For example, a queue with no consumers that has grown until the free disk space drops below a configured limit (50 MB by default). This triggers an alarm and all producers are blocked. Once the messages of the queue are consumed or the queue is purged, the alarm will be cleared, producers unblocked and the spring-cloud-broker app will start working again.
In order to avoid this issue in the future there are two recommendations to follow:
If it's working and you confirm the SCS worker didn't receive the message by checking worker logs, the problem is with the RabbitMQ cluster.
Open the RabbitMQ Management UI as explained here, and check the status of the cluster. It's highly probable that connections are being blocked due to a disk or memory alarm being triggered. Identify the cause of the resource being overused and release it.
For example, a queue with no consumers that has grown until the free disk space drops below a configured limit (50 MB by default). This triggers an alarm and all producers are blocked. Once the messages of the queue are consumed or the queue is purged, the alarm will be cleared, producers unblocked and the spring-cloud-broker app will start working again.
In order to avoid this issue in the future there are two recommendations to follow:
- If you are using a RabbitMQ Pre-Provisioned service plan with Spring Cloud Services, then switch to an On-Demand RabbitMQ service plan. This will prevent other clients from filling up any resource.
- The other is to upgrade to SCS 3.x where there is no longer a dependency on RabbitMQ.
Feedback
Yes
No
Powered by
