RabbitMQ: Options to improve throughput while ensuring messages are delivered in order.

Products

• Services Suite

Issue/Introduction

In enterprise environments, maintaining message ordering while scaling throughput is a typical use case. Standard ordered delivery often relies on a Single Active Consumer (SAC) with a prefetch count of 1. While this guarantees strict ordering, it creates a significant performance bottleneck by limiting processing to a single message at a time per queue.

 

Environment

All Supported RabbitMQ versions

Resolution

A common approach to scaling out queues involves using sharding to increase parallelism, and thereby, throughput.

Super-Streams provides a way to scale out by partitioning a large stream into smaller streams. It integrates with Single Active Consumer(SAC) to preserve message order within a partition. 

If Super-Streams aren't the right fit, exchange types offer similar "routing-based" scaling:

• Consistent Hash Exchange uses hashing to distribute routing keys among queues, ensuring that all messages with the same routing key will go to the same queue.Using the consistent hash exchange requires more upfront configuration, which may be complex for some scenarios.
• x-modulus-hash-exchange  binds N quorum queues to an x-modulus-hash exchange instance (the binding keys do not matter) and uses the Single Active Consumer (SAC) feature on each quorum queue to achieve message ordering, concurrent consumption and fault tolerance .
• New Murmur 3 exchange  implements the same murmer3 hash used by Super Stream clients to ensure message ordering with Super Streams.