Override Policy in RabbitMQ to recover unsynchronized queues

Products

• RabbitMQ

Issue/Introduction

In RabbitMQ clusters, Classic Mirrored Queues (CMQs) can occasionally fall into an unsynchronized state due to node failures or network partitions. When this occurs, the queue stops processing messages, fails to drain, and may display question marks (?) for its state in the RabbitMQ Management UI.

Because multiple queues often share a single High Availability (HA) policy, you cannot simply restart the cluster without risking wider disruption. The safest and most optimal way to recover a locked queue is to apply a temporary, high-priority Override Policy. This forces the problematic queue to drop its mirrors and run on a single node, which clears the deadlock. Once the queue is healthy and processing messages again, the override is removed, allowing the original HA policy to seamlessly rebuild the mirrors on the peer nodes.

Resolution

Run below command(may be from primary node)
rabbitmqctl list_queues name state messages -vhost xxxxxx | grep "xxxxxqueuenamexxxxx"

If you could see the details for queue, you could try policy overriding to sync the queues.

Since all of the queues are covered by one policy, an override is an optimal way of disabling mirroring for one queue only, then re-enabling it.

1. Apply the Override Policy
   Run this command to temporarily strip the mirrors from the locked queue.
   rabbitmqctl set_policy Queue-Override "^xxxxxqueuenamexxxxx$" '{"ha-mode":"exactly", "ha-params":1}' --vhost "xxxxxx" --priority 100 --apply-to classic_queues

2. Verify the Queue
   Wait about 30 to 60 seconds. Check the Management UI or run:
   rabbitmqctl list_queues name state messages -vhost xxxxxx | grep "xxxxxqueuenamexxxxx"

You should see the question marks (?) disappear, the state return to running, and the 150 msgs/s finally begin to drain.

3. Clear the Override Policy
   Once the queue is healthy and processing messages again, remove the temporary policy so your default ha-all policy takes over. This will instruct RabbitMQ to slowly start rebuilding the mirrors on the other two nodes.
   rabbitmqctl clear_policy Queue-Override --vhost "xxxxxx"

The queue will be matched by the existing policy(ha-all) and will start two mirrors on peer nodes.

Note: 

CMQs have unfixable fundamental issues we have communicated years ago. They were deprecated formally in August 2021, the doc guide is very visibly warns about that https://www.rabbitmq.com/docs/3.13/ha

We recommend to implement quorum queues to avoid these issues.