The pre-stop script of rabbitmq-server instance failed with error "Command 'await_online_synchronized_mirror' not found"

Products

• VMware Tanzu Application Service

Issue/Introduction

After Tanzu RabbitMQ on Cloud Foundry tile was upgraded to v10.0.0, the pre-stop script of rabbitmq-server instance might fail with error "Command 'await_online_synchronized_mirror' not found" as show below.

# bosh task logs
Task 4590601 | 07:15:22 | Updating instance rabbitmq-server: rabbitmq-server/c99cc405-xxxx-xxxx-xxxx-9be2887f44b2 (0) (canary)
Task 4590601 | 07:15:23 | L executing pre-stop: rabbitmq-server/c99cc405-xxxx-xxxx-xxxx-9be2887f44b2 (0) (canary) (00:00:04)
                        L Error: Action Failed get_task: Task 8e76bc3b-xxxx-xxxx-xxxx-c5ba266c95d0 result: 1 of 1 pre-stop scripts failed. Failed Jobs: rabbitmq-server.

#pre-stop logs
rabbitmq-server/c99cc405-xxxx-xxxx-xxxx-9be2887f44b2:/var/vcap/sys/log/rabbitmq-server# cat pre-stop.stderr.log
Command 'await_online_synchronized_mirror' not found.
Usage
rabbitmq-upgrade [--node <node>] [--timeout <timeout>] [--longnames] [--quiet] <command> [<command options>]
Available commands:
Help:
   help                          Displays usage information for a command
Upgrade:
   await_online_quorum_plus_one  Waits for all quorum queues and streams to have an above minimum online quorum. This makes sure that no queues/streams would lose their quorum if the target node is shut down
   drain                         Puts the node in maintenance mode. Such nodes will not serve any client traffic or host any primary queue replicas
   post_upgrade                  Runs post-upgrade tasks
   revive                        Puts the node out of maintenance and into regular operating mode. Such nodes will again serve client traffic and host primary queue replicas
Use 'rabbitmq-upgrade help <command>' to learn more about a specific command

 

Environment

Tanzu RabbitMQ on Cloud Foundry 10.0

Cause

The pre-stop script will run the following commands if the Wait for Queue Synchronization option is enabled on RabbitMQ tile Settings for the plan associated with the RabbitMQ service instance. 

rabbitmq-upgrade await_online_quorum_plus_one -t 3600
rabbitmq-upgrade await_online_synchronized_mirror -t 3600

Command rabbitmq-upgrade await_online_synchronized_mirror is for queues using Classic Queue Mirroring feature. However, that feature has been deprecated for a long time and is completely removed since RabbitMQ version 4.x. More details are available on RabbitMQ document. As a result, await_online_synchronized_mirror sub-command is also removed from rabbitmq-upgrade command.

As indicated in the release note, Tanzu RabbitMQ on Cloud Foundry 10.0.0 is shipped with RabbitMQ version 4.0.3. That's why the error occurs. 

Resolution

The product team is working on the fix, which is to remove command rabbitmq-upgrade await_online_synchronized_mirror from pre-stop script. 

Before the new release including the fix is ready, you can also try the following approaches to workaround the issue.

Use --skip-drain flag for BOSH command

The "--skip-drain" flag will bypass running pre-stop script for a BOSH command like bosh deploy or bosh recreate. 

 

Disable Wait for Queue Synchronization 

Disable Wait for Queue Synchronization option for the plan on RabbitMQ tile Settings and do "Apply Changes" for RabbitMQ tile and run upgrade-all-service-instances errand.

If the issue still remains, then the upgrade operation should be performed for individual service instance. Please contact Tanzu support for further assistance in this case.