RabbitMQ: Bindings deleted after a rolling upgrade or node restarts

Products

• RabbitMQ

Issue/Introduction

Bindings are deleted after a rolling upgrade or restart of nodes as a result of which client connections become stale and subscribers are not able to process messages. Re-importing definition files also doesn't work.

 

Cause

A number of issues related to lost and inconsistent transient and semi-transient bindings occur in mnesia-based clusters, that are addressed in Khepri. 

With Mnesia, the original metadata store, bindings are stored in two tables, one for durable bindings (between durable exchanges and durable queues or streams) and another for semi-durable and transient ones (where either the queue is transient or both the queue and the exchange are).

When a node was stopped or failed, all non-replicated transient queues on that node were deleted by the remaining cluster peers. Due to high lock contention around these tables with Mnesia, this could take a while. In the case where the restarted (or failed) node came online before all bindings were removed, and/or clients could begin to create new bindings concurrently, the bindings table rows could end up being inconsistent, resulting in obscure "binding not found" errors.

Khepri avoids this problem entirely by only supporting durable entities and using a very different tree-based data model that makes bindings removal much more efficient and lock contention-free. Also note that transient queues will be removed in a future release of RabbitMQ.

 

 

Resolution

Workarounds for this issue are mentioned below.

Short-term - Mnesia users can work around this problem by using quorum queues or durable classic queues and durable exchanges. Their durable bindings will not be removed when a node stops. Queues that are transient in nature can be declared as durable classic ones with a TTL of a few hours.

Long-term - A whole category of issues with binding inconsistency are addressed with the stabilization of Khepri, a new metadata store that uses a tree of nested objects instead of multiple tables. To avail this upgrade to RabbitMQ 4.0.5+ and enable khepri. Khepri is available only as an experimental feature in version 3.13.

This and other closely related issues are described in the release notes.