RabbitMQ nodes failed to start after memory expansion

Products

• VMware Tanzu RabbitMQ
• RabbitMQ

Issue/Introduction

Customer had 3 rabbitmq nodes cluster and they expanded the memory and found rabbitmq node failed to start up.

Below is the error message from the errdump and the rabbitmq log:

Crash dump is being written to: /var/lib/rabbitmq/erl_crash. dump... 
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed 
[os_mon] memory supervisor port memsup): Erlang has closed

There are also the shovel related errors:

({258127,
{error,{rabbitmq_shovel,{{shutdown,
{failed_to_start_child,rabbit_shovel_dyn_worker_sup_sup,
{'EXIT',{{

{badmatch,[]}
,[{mirrored_supervisor,child,2,
[{file,"mirrored_supervisor.erl"},{line,226}]},
{mirrored_supervisor,maybe_start,4,
[{file,"mirrored_supervisor.erl"}
,{line,376}
]},
{mirrored_supervisor,handle_call,3,[{file,"mirrored_supervisor.erl"}
,{line,286}
]},{gen_server2,handle_msg,2,[{file,"gen_server2.erl"}
,line,1035}
]},{proc_lib,init_p_do_apply,3,[

{file,"proc_lib.erl"}
,{line,329}
]}]},{gen_server2,call,[<0.1959349.0>,{start_child,{{<<"Vhost">>,<<"SHOVEL_xxx_TO_yyy">>},{rabbit_shovel_dyn_worker_sup,start_link,[

{<<"xxx">>,<<"SHOVEL_xxx_TO_yyy">>}
,[{<<"dest-uri">>,[{encrypted

Cause

Every shovel has a URI in its state. That URI contains a password or a JWT token. In order to avoid logging such values when a shovel fails (e.g. to connect), that value is encrypted using https://github.com/rabbitmq/credentials-obfuscation, which uses a node-local one-off encryption key.

The URI is then decrypted right before it is passed on to the AMQP 0-9-1 or AMQP 1.0 client.

The key in question is node-specific. A shovel that has stored some encrypted state on one node cannot be started on another node.

Another issue is that a shovel can start before the key is generated. Shovels boot in parallel to the encryption library.

Resolution

Disabling the plugin and re-creating all shovels will make sure all nodes use the "modern" way of obfuscating credentials.