RabbitMQ federated Quorum Queues with small prefetch count, frequently stop consuming
Article ID: 297329
Updated On:
Products
Support Only for OpenSource RabbitMQ
Issue/Introduction
RabbitMQ federated queues frequently stop consuming when quorum queues with shorter prefetch values are used on the consumers. To recover, the federated queue links need to be restarted.
Most use of federation queues with quorum queues will not have this problem. Only whre small prefetch values are used. By design, federated queues continuously pause and resume and when they are used with quorum queues with a shorter pre-fetch value on consumers, this issue may be triggered. This issue doesn’t happen with federation exchanges used with classic queues, which by default use a larger pre-fetch value.
This affects all current versions of RabbitMQ, including OSS and VMware distributions.
Most use of federation queues with quorum queues will not have this problem. Only whre small prefetch values are used. By design, federated queues continuously pause and resume and when they are used with quorum queues with a shorter pre-fetch value on consumers, this issue may be triggered. This issue doesn’t happen with federation exchanges used with classic queues, which by default use a larger pre-fetch value.
This affects all current versions of RabbitMQ, including OSS and VMware distributions.
Resolution
Technically, this issue could occur without federation. However, it's triggered by fast subscribe-cancel operations that most applications won't perform. Federation doesn't intentionally do this either but in certain circumstances, it might.
This is a product issue and is resolved in RabbitMQ version 3.12.4.
The problem is that a quorum queue could incorrectly calculate the next message for the subscription if the previous subscription was canceled very shortly before (since there were some in-flight messages from the previous subscription). In most cases it doesn't matter because consumers are long-lived and if they get disconnected, it takes a few seconds or more to establish a new connection. In this case, however, the federation could keep the channel open but subscribe and cancel the subscription many times per second. This is because it cancels the subscription when the downstream queue is not empty (because local messages should have a "higher priority"). But if the consumers have a low prefetch, we end up in this loop:
To resolve the issue, please upgrade to RabbitMQ 3.12.4.
Workarounds:
RabbitMQ Quorum Queues
Federation Plugin
Consumer Prefetch
Quorum queues and queue federation do not get along #9138
This is a product issue and is resolved in RabbitMQ version 3.12.4.
The problem is that a quorum queue could incorrectly calculate the next message for the subscription if the previous subscription was canceled very shortly before (since there were some in-flight messages from the previous subscription). In most cases it doesn't matter because consumers are long-lived and if they get disconnected, it takes a few seconds or more to establish a new connection. In this case, however, the federation could keep the channel open but subscribe and cancel the subscription many times per second. This is because it cancels the subscription when the downstream queue is not empty (because local messages should have a "higher priority"). But if the consumers have a low prefetch, we end up in this loop:
To resolve the issue, please upgrade to RabbitMQ 3.12.4.
Workarounds:
- As the issue is triggered by queue federation of quorum queues, use the classic queues instead of quorum queues
- A higher prefetch on the consumer allows more messages to be immediately delivered and the queue remains empty, which means the federation keeps the subscription with the upstream queue. So Increasing the prefetch on the consumers will lower the odds of triggering the issue
RabbitMQ Quorum Queues
Federation Plugin
Consumer Prefetch
Quorum queues and queue federation do not get along #9138
Feedback
Yes
No
Powered by
