RabbitMQ Tile Upgrade from 2.3.5 to 2.4.2 Stuck Due to Khepri DB Errors
search cancel

RabbitMQ Tile Upgrade from 2.3.5 to 2.4.2 Stuck Due to Khepri DB Errors

book

Article ID: 413385

calendar_today

Updated On:

Products

Services Suite VMware Tanzu RabbitMQ RabbitMQ VMware Tanzu Data Intelligence VMware Tanzu Data Suite VMware Tanzu Data Suite

Issue/Introduction

During the upgrade from 2.3.5 to 2.4.2, the following error message appears repeatedly in the RabbitMQ server logs:

[warning] <x.xxx.x> Mnesia->Khepri fallback handling: Mnesia function failed <number> times.
Possibly an infinite retry loop; trying one last time.

The upgrade process then halts and does not complete automatically.

Environment

RabbitMQ Tile 2.3.5 → 2.4.2

Cause

The issue is triggered by the presence of the exchange logger backend configuration, which conflicts with internal Khepri DB initialization logic during the upgrade.

Problematic configuration:

{exchange, [ {enabled, true} ]}

This configuration, when present, leads to repeated Mnesia-to-Khepri fallback failures and stalls the upgrade.

Resolution

Before upgrading, remove or disable the exchange logger backend configuration from the RabbitMQ configuration file.

Remove the following block:

{exchange, [
    {enabled, true}
]}

Once removed, reattempt the upgrade from 2.3.5 to 2.4.2. The upgrade should complete without any further issues.

 

Alternative Approach

  • Monit stop/start: If the upgrade is already stuck, perform the following manual steps to complete it successfully:
    • Stop the RabbitMQ server process on all nodes:
      monit stop rabbitmq-server
    • Start the RabbitMQ server process one node at a time: 
      monit start rabbitmq-server
  • Two step Upgrade: If configuration modification is not feasible, perform a two-step upgrade instead:
    • Upgrade from 2.3.5 → 2.4.0
    • Then upgrade from 2.4.0 → 2.4.2

This approach has been observed to avoid the issue entirely.

Additional Information

Known Issue — Manual workaround available.
Future RabbitMQ Tile release will have the fix.