After Tanzu RabbitMQ on Cloud Foundry tile was upgraded to v10.0.0, the pre-stop script of rabbitmq-server instance might fail with error "Command 'await_online_synchronized_mirror' not found" as show below.
# bosh task logs
Task 4590601 | 07:15:22 | Updating instance rabbitmq-server: rabbitmq-server/c99cc405-xxxx-xxxx-xxxx-9be2887f44b2 (0) (canary)
Task 4590601 | 07:15:23 | L executing pre-stop: rabbitmq-server/c99cc405-xxxx-xxxx-xxxx-9be2887f44b2 (0) (canary) (00:00:04)
L Error: Action Failed get_task: Task 8e76bc3b-xxxx-xxxx-xxxx-c5ba266c95d0 result: 1 of 1 pre-stop scripts failed. Failed Jobs: rabbitmq-server.
#pre-stop logs
rabbitmq-server/c99cc405-xxxx-xxxx-xxxx-9be2887f44b2:/var/vcap/sys/log/rabbitmq-server# cat pre-stop.stderr.log
Command 'await_online_synchronized_mirror' not found.
Usage
rabbitmq-upgrade [--node <node>] [--timeout <timeout>] [--longnames] [--quiet] <command> [<command options>]
Available commands:
Help:
help Displays usage information for a command
Upgrade:
await_online_quorum_plus_one Waits for all quorum queues and streams to have an above minimum online quorum. This makes sure that no queues/streams would lose their quorum if the target node is shut down
drain Puts the node in maintenance mode. Such nodes will not serve any client traffic or host any primary queue replicas
post_upgrade Runs post-upgrade tasks
revive Puts the node out of maintenance and into regular operating mode. Such nodes will again serve client traffic and host primary queue replicas
Use 'rabbitmq-upgrade help <command>' to learn more about a specific command
Tanzu RabbitMQ on Cloud Foundry 10.0
The pre-stop script will run the following commands if the Wait for Queue Synchronization option is enabled on RabbitMQ tile Settings for the plan associated with the RabbitMQ service instance.
rabbitmq-upgrade await_online_quorum_plus_one -t 3600
rabbitmq-upgrade await_online_synchronized_mirror -t 3600
Command rabbitmq-upgrade await_online_synchronized_mirror
is for queues using Classic Queue Mirroring feature. However, that feature has been deprecated for a long time and is completely removed since RabbitMQ version 4.x. More details are available on RabbitMQ document. As a result, await_online_synchronized_mirror
sub-command is also removed from rabbitmq-upgrade
command.
As indicated in the release note, Tanzu RabbitMQ on Cloud Foundry 10.0.0 is shipped with RabbitMQ version 4.0.3. That's why the error occurs.
The product team is working on the fix, which is to remove command rabbitmq-upgrade await_online_synchronized_mirror
from pre-stop script.
Before the new release including the fix is ready, you can also try the following approaches to workaround the issue.
Use --skip-drain flag for BOSH command
The "--skip-drain" flag will bypass running pre-stop script for a BOSH command like bosh deploy
or bosh recreate
.
Disable Wait for Queue Synchronization
Disable Wait for Queue Synchronization option for the plan on RabbitMQ tile Settings and do "Apply Changes" for RabbitMQ tile and run upgrade-all-service-instances errand.
If the issue still remains, then the upgrade operation should be performed for individual service instance. Please contact Tanzu support for further assistance in this case.