Migrating SCDF and TDF Streams from Mirrored Classic Queues to Quorum Queues for Tanzu RabbitMQ

Products

VMware Tanzu RabbitMQ VMware Tanzu Spring Essentials

Issue/Introduction

When upgrading the Tanzu RabbitMQ tile from version 2.4.x to 10.x (RabbitMQ 4.0.x), the upgrade-all-service-instances errand or manual service instance updates may fail with the following error in the startup_stderr.log:

BOOT FAILED: {error, {failed_to_deny_deprecated_features, [classic_queue_mirroring]}}

This occurs because Tanzu RabbitMQ 10.x removes support for Classic Mirrored Queues. Any existing mirroring policies or mirrored queues will prevent the RabbitMQ node from starting.

Environment

Spring Cloud Data Flow (SCDF) for Cloud Foundry instance using Classic Mirrored Queues for high availability
Tanzu Data Flow (TDF) Instance Classic Mirrored Queues for high availability
Tanzu RabbitMQ on Tanzu Platform 10.0 (RabbitMQ tile 4.0) and later

Cause

Spring Cloud Data Flow (SCDF) instances often use Classic Mirrored Queues for high availability. In RabbitMQ 4.0 (included in Tanzu RabbitMQ tile 10.x), mirroring has been removed. Users must migrate these queues to Quorum Queues to maintain high availability or revert them to non-replicated classic queues.

Resolution

To ensure a zero-data-loss migration and maintain high availability, follow these steps to migrate SCDF backing queues to Quorum Queues before upgrading the tile.

Prerequisites

Verify that all SCDF and TDF streams and task applications are updated to Spring Boot 3.0+.
Export current definitions as a backup: rabbitmqctl export_definitions /path/to/definitions.json.

Step 1: Identify Existing Mirrored Queues and Policies

Review the commands before running them:

rabbitmq-diagnostics list_vhosts
rabbitmq-diagnostics list_queues --vhost <vhost_name>
rabbitmq-diagnostics list_policies --vhost <vhost_name>

Or via the VMware Tanzu RabbitMQ dashboard URL.

Identify any policies containing ha-mode, ha-params, or ha-sync-mode.

Step 2: Stop Traffic and Clear Policies

Stop Publishers: Pause all data entry into the streams.
Undeploy SCDF Streams: Undeploy the streams to stop consumer activity.
Verify Message Counts: Ensure all queues have 0 messages.
Remove Mirroring Policies:

This command will make changes to your system. Review it carefully before running:

rabbitmqctl clear_policy [-p vhost] <policy_name>
# OR
rabbitmqctl -q remove_classic_queue_mirroring_from_policies

Or via the VMware Tanzu RabbitMQ dashboard URL.

Step 3: Delete Old Classic Queues

This command cannot be undone. Verify every parameter before running:

# Use the Management UI or rabbitmqadmin to delete the existing classic queues associated with the streams.
rabbitmqadmin --vhost="vhost-name" delete queue name="queue-name"

Or via the VMware Tanzu RabbitMQ dashboard URL.

Step 4: Redeploy SCDF Streams with Quorum Queues

For Pre-package application redeploy the streams using the quorum.enabled=true property to ensure the new queues are created as Quorum Queues.

Review this command before running it:

stream deploy <my-stream> \
        --properties "app.*.spring.cloud.stream.rabbit.bindings.input.consumer.quorum.enabled=true, \
                      app.*.spring.cloud.stream.rabbit.bindings.output.producer.quorum.enabled=true"

note: These properties can alternatively be managed via a Config Server

For custom streams with explicit queue creation, see Implementing VMware Tanzu RabbitMQ Quorum Queues in Spring AMQP

Step 5: Final Verification and Tile Upgrade

Verify that all new queues are listed as quorum in the Management UI or via rabbitmq-diagnostics list_queues.
Test Stream Applications

Step 6: RabbitMQ Tile Upgrade

Run the Tanzu RabbitMQ tile upgrade to version 10.x (optional)