RabbitMQ upgrade from Tanzu RabbitMQ Tile 2.4.2 to 2.4.3 hangs with federation
Article ID: 426156
Updated On:
Products
Issue/Introduction
RabbitMQ tile upgrade from version 2.4.2 to 2.4.3 hangs after one node is upgraded and remains in progress for many hours.Cancelling the task and re-running the upgrade results in the same stuck state.
During the upgrade, the following errors are observed in the RabbitMQ logs.
[error] Error on AMQP connection XXX - Federation link (upstream: test, policy: testp), user: 'federation', state: opening):
[error] access to vhost 'vhost' refused for user 'federation'
[warning] Federation exchange 'EEE' in vhost 'vhost' did not connect to exchange 'EEE' in vhost 'vhost' on amqps: YYY/vhost. Reason: {error, AAA [warning] <BBB> not_allowed}
(Where XXX, YYY, AAA, and BBB represent redacted values.)
Environment
- Tanzu RabbitMQ Tile: 2.4.2 → 2.4.3
- RabbitMQ: 3.12.15
- Erlang: 25
Cause
Active RabbitMQ federation links during the tile upgrade caused the upgrade task to hang after one node was upgraded.
Resolution
This is a product issue triggered when federation remains enabled during the node upgrade process, and is planned to be fixed in a future Tanzu RabbitMQ Tile release.Subscribe to this KB article to receive updates when a fix and target version are available.
Workaround
Use one of the following workarounds during the upgrade.
- Temporarily disable all RabbitMQ federation links before starting the tile upgrade, then re-enable them after the upgrade completes.
- Identify nodes hosting the most federation links and restart those nodes to clear the federation state before proceeding with the upgrade, then re-run the upgrade.
Note: Disabling federation may temporarily impact cross-cluster or cross-vhost message flow, so plan a maintenance window accordingly.
Additional Information
Feedback
