Required Feature Flags when Upgrading RabbitMQ to 3.11 or 3.12
search cancel

Required Feature Flags when Upgrading RabbitMQ to 3.11 or 3.12

book

Article ID: 297355

calendar_today

Updated On:

Products

Support Only for OpenSource RabbitMQ

Issue/Introduction


The goal of this article is to increase awareness and visibility of RabbitMQ's required feature flags to help prepare our customers for a smooth upgrade experience. This article will cover which feature flags are required in which versions, how to check their status, and how to enable them for any of our RabbitMQ offerings (OSS, TAS, K8s).

Feature flags are a well documented topic and are used to enable specific features within RabbitMQ and to establish compatibility within a mixed-version cluster. When preparing to upgrade to RabbitMQ 3.11 or 3.12, there are certain feature flags that were introduced in previous versions that will be required to be enabled prior to upgrading to 3.11 or 3.12.

Below are links outlining this requirement and the specified feature flags required for each version:
 
Without having the required flags enabled prior to upgrading a node will not be able to start up properly; essentially causing the upgrade to fail. If you've inadvertently upgraded without meeting this requirement you will see log messages like the following; stating which flag was required but not enabled, thus preventing a successful boot: 
2022-07-13 11:29:28.366877+02:00 [error] <0.232.0> Feature flags: `implicit_default_bindings`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
2022-07-13 11:29:28.366905+02:00 [error] <0.232.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
2022-07-13 11:29:28.366905+02:00 [error] <0.232.0>                                               implicit_default_bindings}
2022-07-13 11:29:28.372830+02:00 [error] <0.232.0>
2022-07-13 11:29:28.372830+02:00 [error] <0.232.0> BOOT FAILED
2022-07-13 11:29:28.372830+02:00 [error] <0.232.0> ===========
2022-07-13 11:29:28.372830+02:00 [error] <0.232.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
2022-07-13 11:29:28.372830+02:00 [error] <0.232.0>
It is important to enable these before upgrading as downgrading is not fully supported and will cause extra work to fix. If you are in this position the only way to address it is to uninstall RabbitMQ, reinstall the previous version, enable the required feature flags, then proceed with the initial upgrade again.

The best practice and recommendation to handle this requirement is to always enable all feature flags after a successful rolling upgrade so you are already prepped for the next upgrade. Enabling feature flags costs little to no overhead and will ensure a smooth transition to later versions.

Below are steps you can perform on any RabbitMQ offering to check feature flag status and/or enable them.

Environment

Product Version: 3.11
OS: Any

Resolution


Whether you are using OSS RabbitMQ, RabbitMQ for Tanzu Application Service, or RabbitMQ for Kubernetes checking and enabling feature flags can only be done through CLI tools or the Management UI. As we improve feature flag maintenance, there is a TAS version and K8s tool where this will be more automated (details on this below).

If you find yourself in a position where you need to manually check/enable feature flags, your options are the following:

CLI

  • To list feature flags: 
    rabbitmqctl list_feature_flags
  • To enable a feature flag (or all currently disabled flags): 
    rabbitmqctl enable_feature_flag <all | name>

 

Management UI

It is also possible to check the status and enable feature flags from the Management plugin UI.

  • Management UI > Admin tab > Feature Flags tab > Click the Enable button under the State column:

image.png


OSS RabbitMQ

With OSS RabbitMQ  your goal should be to get to version 3.10 where all previous feature flags are present and can be enabled before 3.11+ where they are required. Once on version 3.10, enable all feature flags, upgrade to 3.11, enable all feature flags again, and finally proceed to upgrade to 3.12.

RabbitMQ for TAS

Beginning with RabbitMQ for TAS version 2.2.4, all feature flags will be automatically enabled when running the upgrade-all-on-demand-instances ("Upgrade All Service Instances") errand, eliminating the need to handle them manually or step through tile versions (discussed below). Note, if this errand is disabled and/or otherwise not run, the feature flags will not be enabled properly, so this errand must always be run when a tile upgrade or patch is installed, otherwise the patch will not apply to the instances.

If you are on a RabbitMQ for TAS version prior to 2.2.4, the tile will automatically enable all feature flags after each apply changes; but you will need to step through tile versions similarly to OSS RabbitMQ instructions above. To do this you will need to ensure that you select the RabbitMQ 3.10 tile version, upgrade all service instances, select 3.11 tile version, upgrade all service instances, select 3.12 tile version, and perform a final upgrade all service instances.

The above instructions are the preferred method to handle this requirement, but you can still use CLI or Management UI to do the same manually.

RabbitMQ for K8s

RabbitMQ for K8s will follow the same instructions as OSS RabbitMQ detailed above, unless you have Cluster Operator installed. If you have Cluster Operator installed this tool will automatically list and enable all available feature flags on cluster start: 
Cluster Operator does not support disabling any RabbitMQ feature flags. The Operator lists all available feature flags and enables all of them at cluster start.

Note: Once feature flags are enabled they cannot be disabled. This is by design.

It is my hope that this will help clarify how to properly handle feature flags in RabbitMQ, however, if you encounter any issues or have any questions please reach out to us with a support request and we'll be happy to assist you.

Powered by Wolken Software