"consumer ack timed out..." error on client application after upgrading RabbitMQ broker

Products

• VMware RabbitMQ

Issue/Introduction

You upgraded your RabbitMQ broker and now your consumer applications are throwing an error similar to the following below:

2021-08-30 23:37:34.116 [error] Channel error on connection (10.85.216.29:52948 -> 10.85.106.163:5672, vhost: 'vhost1', user: 'user1'), channel 3: operation none caused a channel exception precondition_failed: consumer ack timed out on channel 3

Environment

Product Version: 2.0

Resolution

The following are recent changes to consumer_timeout in RabbitMQ:
 

OSS version	consumer_timeout change
v3.8.2	consumer_timeout was made configurable through new style format. The default value is infinity.
v3.8.15	consumer_timeout default value was changed to 15 minutes. The previous versions default value was infinity.
v3.8.17	consumer _timeout default value was increased to 30 minutes.

You can configure your custom consumer_timeout through the RabbiMQ broker configuration.

The "consumer ack timed out" error means that your application (consumer) failed to acknowledge a message within a time period that is defined by consumer_timeout. 

To work around this issue, consider either of the following suggestions:

• The default values usually does suffice and we recommend reviewing your consumer code. Analyze if there is any processing that is causing a delay and messages not to be acknowledge in a 15 minute or 30 minutes timeout. You might want to check your bottlenecks and consumer utilisation.
• Increase consumer_timeout to the longest time that the consumer application acknowledges a message by specifying the Delivery Acknowledgement Timeout on the configuration file. You need to restart RabbitMQ node/broker for changes to take effect. If you are using Tanzu RabbitMQ for VMs, refer  to Expert Mode - Overriding RabbitMQ Server Configuration to override default configuration.