RabbitMQ's abrupt termination by Systemd causes delays in node startup time
Article ID: 392573
Updated On:
Products
Issue/Introduction
RabbitMQ's abrupt termination by Systemd and its repeated restarts delay the recovery process.
Example log events from RabbitMQ server logs and system logs.
# Systemd stops rabbitmq
Rabbit-test systemd[1]: Stopping RabbitMQ broker...
Rabbit-test rabbitmqctl[xxxxxxx]: Shutting down RabbitMQ node
# Abrupt termination by systemd
Rabbit-test systemd[1]: rabbitmq-server.service: Stopping timed out.
Terminating.
# Systemd tries to start RabbitMQ
Rabbit-test systemd[1]: Starting RabbitMQ broker...
# Systemd times out starting Rabbit service
Rabbit-test systemd[1]: rabbitmq-server.service: Failed with result 'timeout'
# Rabbit asked to shut down due to systemd timeout
[notice] SIGTERM received - shutting down
Environment
Applicable to all the supported RabbitMQ and Erlang versions.
Cause
Once RabbitMQ is abruptly terminated, it rebuilds all indices during the next startup as part of the recovery procedure. This process can take a considerable amount of time. The Systemd repeatedly enters a cycle of abruptly terminating and restarting RabbitMQ every 10 minutes (default value for Systemd timeout) until the recovery procedure is completed. After this, RabbitMQ must recover all queues and vHosts, a process that can take potentially hours, depending on the RabitMQ environment.
Resolution
This is an environment-specific issue. The default value for Systemd timeout (TimeoutStartUSec), may be insufficient for nodes with large queues or slow consumers. The issue can be resolved by Increasing and adjusting the Systemd Timeout (TimeoutStartUSec) in the RabbitMQ systemd unit file to allow more time for graceful shutdown.
This can be done either by scripting the creation of the override file or can using systemctl edit rabbitmq-server.service to create one manually.
Additional Information
Feedback
