RabbitMQ Queue Crash After Unclean Shutdown on Single-Node Setup

Products

• VMware Tanzu Data Suite

Issue/Introduction

In a single-node RabbitMQ deployment, a queue may fail to start after the broker restarts. This often appears as queue-related errors or forced connection closures during application startup or normal operation.

Example symptoms:

Queue fails to initialize with errors referencing not_found or crashed and failed to restart

Clients receive:

connection_forced: "broker forced connection closure with reason 'shutdown'"

Cause

This issue typically occurs due to an unclean shutdown of the RabbitMQ broker, such as:

• Power failure

• Forced process termination (kill -9)

• System crash or unexpected reboot

• Out-of-memory (OOM) conditions

On startup, RabbitMQ detects the unclean shutdown and attempts to recover queues and persisted messages. If a queue's data was corrupted (e.g., due to an incomplete write), it may fail to load.

Typical log entries include:

 

 dropped 0/0/0 persistent messages and 0 transient messages after unclean shutdown exception exit: {function_clause, [{rabbit_msg_store, reader_pread_parse, [[eof]]... 

These messages confirm that a queue crashed due to unreadable message files caused by the prior unclean shutdown.

Resolution

To reduce risk of queue crashes or data loss:

• Use mirrored or quorum queues for high availability.

• Avoid forced shutdowns (kill -9) — always stop RabbitMQ cleanly:

  systemctl stop rabbitmq-server

Additional Information

If RabbitMQ version is 3.13 or later,  the issue could be as described in the following GitHub reference. https://github.com/rabbitmq/rabbitmq-server/pull/14576/files