RabbitMQ Quorum queues with large backlogs become unresponsive and reject new consumers.
Article ID: 422254
Updated On:
Products
Issue/Introduction
In RabbitMQ 4.1.4, quorum queues (QQs) with very large backlogs—such as millions of messages or large payloads—can become unresponsive. Management UI actions like "Get message" or "Purge" hang or fail, and new consumers time out on basic.consume. This occurs even without active consumers, leaving the queue stalled despite appearing present.
Typical symptoms include:
- The queue shows a huge size, but no progress or consumer activity.
- AMQP errors: operation basic.consume caused a connection exception internal_error: "timed out consuming from quorum queue." followed by connection closure warnings.
- Other queues and nodes remain responsive.
Environment
- RabbitMQ version 4.1.4
- Quorum queues with extreme backlogs (millions of messages/large payloads)
Cause
This is a known product issue in RabbitMQ 4.1.4 that affects quorum queue Raft processes during heavy load, restarts, or leadership transfers, causing leader/follower crashes or timeouts. This leaves the queue in a noproc or non-progressing state, blocking all operations without self-recovery.
Resolution
Upgrade to RabbitMQ 4.2.0 or later, which fixes Raft handling and adds gradual leadership transfers for overloads/restarts. Test in staging first, as quorum queue metrics and behavior changed (update dashboards/alerts).
Additional Information
Workarounds:
- Delete and recreate the affected queue once stalled.
- For data preservation: Stop publishers, create a replacement queue, reroute traffic, then delete the stuck one.
Proactive measures:
- Limit backlogs with adequate consumers, prefetch limits, and dead-lettering.
- Monitor large idle queues for management operation failures.
References:
Feedback
