RabbitMQ: Handling 'Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED..'

Products

• VMware Tanzu RabbitMQ

Issue/Introduction

This article describes actions to take and best practices to follow to handle the well-documented channel error logged below.

Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 2 timed out. Timeout value used: 1800000 ms.

Resolution

Workarounds 

•  Restart the consumer apps which will close all connections and rule out network connectivity issues.

Best practices to consider to address this scenario are listed below.

Consumer applications:

• The consumer app can be modified to add a handler or callback to handle the channel exception and automatically restart the consumer apps.  The implementation would be specific to the client-libraries used. See links for spring-amp and java client libraries below. 
  • Spring-amqp
  • Java client library
• For manual acknowledgements, messages should be batched and acknowledged in the correct order.
• Acknowledge ( basic.Ack) after you have processed messages, and not before.
• Use Micrometer to enable tracing on the consumer side, if it is supported by the application language/framework.

Cluster :

• Use Prometheus metrics to monitor queues/channels/disks and set up alerts.
• Durable or quorum queues will write to disk, which could result in disk alarms, if messages are not consumed. To address this, provision disks for worst case scenarios.
• Use separate connections for publishers and consumers, so that when publisher blocks on disk alarms it doesn't impact consumers.