RabbitMQ 3.13 Erlang 26 TLS Verification Default Change

Products

Pivotal RabbitMQ Support Only for OpenSource RabbitMQ VMware RabbitMQ VMware Tanzu RabbitMQ

Issue/Introduction

After upgrading to RabbitMQ 3.13.X and Erlang 26 you notice connection issues with certain components where TLS is enabled like federation links, shovels, and LDAP.

Using TLS-enabled LDAP as an example, you may notice error messages in node logging similar to the following:

LDAP connect error: {error,"connect failed"}

LDAP DECISION: login for username: {error,ldap_connect_error}

HTTP access denied: rabbit_auth_backend_cache failed authenticating username: ldap_connect_error

There can be other causes for the above "ldap_connect_error", but if this is occurring immediately post-upgrade to Erlang 26 it's a strong possibility that this is due to a TLS verification default change detailed below.

Cause

RabbitMQ 3.13.0 will be the first release to begin requiring a minimum version of Erlang 26. Erlang 26 changes the verification default to be enabled from the previous default of disabled:

Change the client default verify option to verify_peer. Note that this makes it mandatory to also supply trusted CA certificates or explicitly set verify to verify_none. This also applies when using the so called anonymous test cipher suites defined in TLS versions pre TLS-1.3.

This notable change was also mentioned in RabbitMQ 3.13.0 release notes to help spread awareness:

Starting with Erlang 26, client side TLS peer certificate chain verification settings are enabled by default in most contexts:
from federation links to shovels to TLS-enabled LDAP client connections.

If using TLS peer certificate chain verification is not practical or necessary, it can be disabled. Please refer to the docs of the feature in question, for example, this one on TLS-enabled LDAP client connections.

Resolution

If you are already using verification with certificates implemented, this change should not affect your cluster, but if you were previously using the disabled default value you may notice this issue post-Erlang 26 upgrade.

As the above release notes state, you will need to either start providing a certificate chain in your configuration, or disable verification to revert to the old, less secure, behavior by setting the following for each affected component (optionally you can disable TLS entirely).

The following ssl_options for verify to be verify_none is what will need to be added to your rabbitmq configurations files (whichever are being used):

{ssl_options, [
    {verify, verify_none}
]}

If you are dealing with TLS enabled LDAP and advanced.config, for example, you would need to add ssl_option for verify_none like so:

[
  {rabbitmq_auth_backend_ldap, [
    {ssl_options, [
      %% Disables peer certificate chain verification.
      %%
      %% Doing so loses one of the key benefits of TLS and make the setup less secure
      %% but also significantly simplifies node configuration.
      {verify, verify_none}
    ]}
  ]}
].

Ref: LDAP Peer Verification

If you are using TLS-enabled LDAP and the rabbitmq.conf file, this same option would look like:

auth_ldap.ssl_options.verify = verify_none

Ref: LDAP TLS

If you are using our RabbitMQ on Cloud Foundry solution then you would perform the same steps as above, then proceed as instructed here: Overriding RabbitMQ Server Configuration

Note: This will essentially revert your verification to Erlang 25 default verification behavior and resolve connection issues moving forward, but please be aware this will need to be configured for all affected components.

If you are dealing with TLS-enabled Federations you will be dealing with Federation URIs to make the verify changes. Please note while setting URI query parameters the ? character is used to begin the query component of the URI. It separates the base URL from any query parameters that may be included in the request. The query component allows the client to send additional data to the server, often in the form of key-value pairs.

In this example: https://www.searchuri.com/search?query=hola&sort=asc&page=2

https://www.searchuri.com/search is the base URL.
The ? indicates that the following part will include query parameters.
query=hola, sort=asc, and page=2 are the query parameters, where query, sort, and page are keys, and hello, asc, and 2 are their respective values.
Multiple query parameters can be included in a URI, separated by the & character.

Example Federation URI config (with TLS enabled using amqps):

eg: amqps://myhost?cacertfile=/path/to/ca_certificate.pem&keyfile=/path/to/client_key.pem&verify=verify_none

If you have any questions or require any assistance with verifying or resolving the above issue, please don't hesitate to reach out to support with a ticket and we'll be happy to assist you.

Additional Information

Helpful Links: