Upgrade All Service Instances errand fails for RabbitMQ 2.2.7

Products

• VMware RabbitMQ

Issue/Introduction

When running the Upgrade All Service Instances errand, it may fail when attempting to upgrade a cluster with the following conditions:

• It's a Multi node cluster
• Shovels have been configured on the cluster

The errand will fail running the post start for RabbitMQ. The following will also be seen in the RabbitMQ server log:

2024-01-02 15:55:18.111453+00:00 [notice] <0.44.0> Application rabbitmq_shovel exited with reason: {{shutdown,{failed_to_start_child,rabbit_shovel_dyn_worker_sup_sup,{'EXIT',{function_clause,[{rabbit_shovel_dyn_worker_sup_sup,id,[<<"shovel_example">>],[{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},{line,100}]},{rabbit_shovel_dyn_worker_sup_sup,'-cleanup_specs/0-fun-2-',2,[{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},{line,90}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,503}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,499}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,495}]},{rabbit_shovel_dyn_worker_sup_sup,cleanup_specs,0,[{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},{line,93}]},{rabbit_shovel_dyn_worker_sup_sup,start_child,2,[{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},{line,42}]},{rabbit_shovel_dyn_worker_sup_sup,'-start_link/0-lc$^0/1-0-',1,[{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},{line,28}]}]}}}},{rabbit_shovel,start,[normal,[]]}}

Environment

Product Version: 2.1

Resolution

RabbitMQ 3.12.8 introduced a change to the id format used by the RabbitMQ shovel supervisor. As a result, when the first node of the cluster is upgraded, it fails due to the new code hitting old ID format stored in mnesia.

https://github.com/rabbitmq/rabbitmq-server/issues/9894

This issue was fixed in OSS 3.12.9

Bug Fixes
Shovels on 3.12.8 nodes failed during a rolling cluster upgrade due to internal
identifier format changes.
 
Starting with this release, both old and new formats are supported for upgrade safety.
 
GitHub issue: #9894

These versions of RabbitMQ for Tanzu Application Service contain the fix:

• 2.2.8+
• 2.3.0+

You will need to upgrade to these versions to resolve the issue.