How to fix lager_error dropped messages on VMware RabbitMQ

Products

• VMware RabbitMQ

Issue/Introduction

You are running Rabbitmq and you notice warning messages, similar to the ones below, appear in the logs. 

[warning] <0.112.0> lager_error_logger_h dropped 42 messages in the last second that exceeded the limit of 1000 messages/sec
[warning] lager_file_backend dropped 32 messages in the last second that exceeded the limit of 50 messages/sec

Environment

Product Version: 3.8

Resolution

The lager warning log messages are expected when there are too many logging messages per second, exceeding beyond 50 messages/second.

Note: Please note that logging messages are different from AMQP messages.

One possible cause for the error message the connection churn being too high, which will make the broker log connect and disconnect.

To resolve this issue, follow the steps below:

• Check if you have high connection churn. You may want to check client's code or logs to see if you observe connecting or disconnecting in a short span of time. 
• If you are seeing normal connection level and you are still receiving these errors, there is a way to configure Lager, the logging library used by RabbitMQ, to increase the default messages/second.

Starting with version in VMware RabbitMQ 3.7, you can configure Lager using the advanced.config file.

Below is a sample configuration:

[

{lager, [
{error_logger_hwm, 300}
]}

].

You can verify a node's error_logger_hwm by issuing the command:

rabbitmqctl eval 'application:get_env(lager, error_logger_hwm).'