Quorum queue leadership shifts due to busy_dist_port in RabbitMQ

Products

• RabbitMQ
• VMware Tanzu RabbitMQ

Issue/Introduction

Issue Leadership for quorum queues unexpectedly migrates to a single broker node in a cluster, even when nodes are healthy and no network partitions exist. This concentration of leadership leads to high publish latency and consumer lag on the overloaded node.

Symptoms

• Quorum leader distribution becomes heavily skewed (e.g., one node hosting >90% of leaders).
• Nodes show consistent uptime with no evidence of restarts.
• Logs contain the following warning:

  <0.243.0> rabbit_sysmon_handler busy_dist_port <0.1782.0> [{name,delegate_management_4}, ...]

• Logs show a "re-election storm" across multiple queues:

  queue '####' in vhost '####': leader -> follower in term: 2queue '####' in vhost '####': detected a new leader {'####','rabbit@node'} in term 2

Environment

• VMware Tanzu RabbitMQ
• RabbitMQ 3.10.x and higher using Quorum Queues

Cause

Nodes communicate in a cluster over the Erlang Distribution Port. If the volume of Raft heartbeats, management traffic, or message synchronization exceeds the RABBITMQ_DISTRIBUTION_BUFFER_SIZE, the port is marked "busy." This delays critical heartbeats, triggering Raft election timeouts and causing leadership to shift to the first available node.

Resolution

1. Verify the current distribution buffer limit by running the following command:

   rabbitmqctl eval 'erlang:system_info(dist_buf_busy_limit).'

2. Increase the buffer size by setting the RABBITMQ_DISTRIBUTION_BUFFER_SIZE environment variable. A recommended value for high-traffic environments is 256000 (256MB).
3. Restart the RabbitMQ node to apply the new environment variable.
4. Rebalance the quorum leaders to restore an even distribution:

   rabbitmq-queues rebalance "all" --vhost ####

   Replace #### with the target vhost name.

Additional Information

See the RabbitMQ Runtime Documentation for further tuning options.

To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.