RabbitMQ quorum queue segment files kept growing

Products

• Pivotal RabbitMQ
• VMware RabbitMQ
• VMware Tanzu RabbitMQ

Issue/Introduction

RabbitMQ quorum queue segment file will keep growing until the queue getting deleted:

Example

• Before publishing the messages to quorum queue

  $ pwd
  /var/lib/rabbitmq/mnesia/[e-mail]/quorum/[e-mail]/##_###############
  $ du -ch *.segment | grep total$
  2.7M    total

• After consecutively publishing 100 messages, with 10k for each to this queue, the quorum queue segment file grows as expected

  $ du -ch *.segment | grep total$
  3.7M    total
  $ ls -ltrh *.segment
  -rw-r--r-- 1 rabbitmq rabbitmq  99K <month> <day> <time> 00000001.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 104K <month> <day> <time> 00000002.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 104K <month> <day> <time> 00000003.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 104K <month> <day> <time> 00000004.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 105K <month> <day> <time> 00000005.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 730K <month> <day> <time> 00000006.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 804K <month> <day> <time> 00000007.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 545K <month> <day> <time> 00000008.segment
  -rw-r--r-- 1 rabbitmq rabbitmq  26K <month> <day> <time> 00000009.segment
  -rw-r--r-- 1 rabbitmq rabbitmq  26K <month> <day> <time> 00000010.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 624K <month> <day> <time> 00000011.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 383K <month> <day> <time> 00000012.segment

• Consume all the messages in the queue, the queue itself became empty however the segment files kept growing(00000013.segment+00000014.segment were newly generated to record the consume operations)

  $ ls -ltrh *.segment
  -rw-r--r-- 1 rabbitmq rabbitmq  99K <month> <day> <time> 00000001.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 104K <month> <day> <time> 00000002.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 104K <month> <day> <time> 00000003.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 104K <month> <day> <time> 00000004.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 105K <month> <day> <time> 00000005.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 730K <month> <day> <time> 00000006.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 804K <month> <day> <time> 00000007.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 545K <month> <day> <time> 00000008.segment
  -rw-r--r-- 1 rabbitmq rabbitmq  26K <month> <day> <time> 00000009.segment
  -rw-r--r-- 1 rabbitmq rabbitmq  26K <month> <day> <time> 00000010.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 624K <month> <day> <time> 00000011.segment
  -rw-r--r-- 1 rabbitmq rabbitmq 425K <month> <day> <time> 00000012.segment
  -rw-r--r-- 1 rabbitmq rabbitmq  26K <month> <day> <time> 00000013.segment
  -rw-r--r-- 1 rabbitmq rabbitmq  25K <month> <day> <time> 00000014.segment

• However, Except for deleting the queue (which will have the queue messages lost), none of the following will help with a immediate cleanup of the segment files:
  • purging the queue
  • dropping the quorum queue member and add it back
  • restarting the node or the cluster.

Environment

RabbitMQ 4.0.5 + Erlang 27.2.2

Cause

This is part of the current design nature of the quorum queue, the segment file will grow, they just shouldn't grow ultimately and get too large.

The quorum queue has its own mechanism to remove the segment file but not simply controlled by user-facing parameters.

The document https://www.rabbitmq.com/docs/quorum-queues#disk-space also recommends to make sure disk over-provisioned in environments with heavy quorum queue usage and/or large messages flowing through them, see <Disk Space> Section of the document 

Resolution

- A temporary workaround is to delete and recreate the quorum queue(that will cause the data lost of the quorum queue, please confirm with the customer)

- There is some improvement in RabbitMQ 4.0.8 to do more frequent checkpoint, see https://github.com/rabbitmq/rabbitmq-server/pull/13587

- Suggest to upgrade to RabbitMQ 4.1.0 and above, which contains further improvements that will significantly mitigate this space consumption concerns.

 

Additional Information

It's hard to immediately cleanup the segment files other than recreating QQ, however, we can mitigate this issue from several perspectives:

1) Avoid sending too many big messages(more than 1M for each) in a batch, or consider if you can compress those big messages before publishing to the queue.

2) Set raft.segment_max_entries to a lower value like 128 in large message scenario: https://www.rabbitmq.com/docs/quorum-queues#performance-tuning-large-messages, however, this would only take effect for the newly created quorum queue, but not for the existing QQ. With lower segment entries you will see more segment files with large message which make it more possible to be efficiently truncated.

3) Upgrade to higher RabbitMQ versions