RabbitMQ client connection reset with message: ForgivingExceptionHandler An unexpected connection driver error occurred

Products

• Support Only for OpenSource RabbitMQ

Issue/Introduction

The client applications encountered connection issues with the RabbitMQ servers with the below error messages.
 

Errors from RabbitMQ Server logs:

[error] <x.yy.z> ** State machine <x.yy.z> terminating

Errors from Client/Consumer logs:
 

ForgivinExceptionHandler An unexpected connection driver error occurred (Exception message: Connection reset)"

This affects all current versions of RabbitMQ, including OSS and VMware distributions.

Resolution

Possible causes of these error messages:

• The client detected an exception in one of the Consumer interface implementation methods, most likely Consumer#handleDelivery. The error says that a TCP connection was closed, but it is not necessarily a connection to RabbitMQ. So, something in the consumer delivery handler tried to perform an operation that threw an I/O exception stating that the other end has closed its end of the TCP connection.
• A  “Connection reset” can also happen due to a network device or part of the virtual environment’s networking stack.
• Finally, this issue can also occur due to a known Erlang product issue, OTP-18476, which  is related to a pipelining/keepalive queue issue, that could result in unexpected "socket_remotly_closed" errors.

To resolve the issue, here are the suggested recommendations.

• Upgrade to the Erlang version 25.3.1 or above, which has the fix for OTP-18476.
• Please make sure that there are no network connectivity issues between the clients and the RabbitMQ that are related to a network device or part of the virtual environment’s networking stack.

References
OTP-25.3.1.README
Troubleshooting Network Connectivity