RabbitMQ Failing to Startup on 7.8.1 with "required feature flag not enabled!"

Products

• Carbon Black EDR

Issue/Introduction

When upgrading to 7.8.1, RabbitMQ fails to startup or times out. In /var/log/cb/rabbitmq/startup_log a message similar to this is found. 

2025-02-05 11:29:28.366877+02:00 [error] <0.232.0> Feature flags: `implicit_default_bindings`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
2025-02-05 11:29:28.366905+02:00 [error] <0.232.0> Failed to initialize feature flags registry: {disabled_required_feature_flag, 2025-02-05 11:29:28.366905+02:00 [error] <0.232.0> implicit_default_bindings}
2025-02-05 11:29:28.372830+02:00 [error] <0.232.0>
2025-02-05 11:29:28.372830+02:00 [error] <0.232.0> BOOT FAILED
2025-02-05 11:29:28.372830+02:00 [error] <0.232.0> ===========
2025-02-05 11:29:28.372830+02:00 [error] <0.232.0> Error during startup:
{error,failed_to_initialize_feature_flags_registry}
2025-02-05 11:29:28.372830+02:00 [error] <0.232.0> 

Environment

• Carbon Black EDR Server: 7.8.1 and higher

Cause

Older EDR servers may not have the feature flags enabled that are required for rabbitmq-server 3.11 and higher. 

Resolution

Run through a single command line. 

1. Run this command to downgrade rabbitmq-server to 3.10 in order to enable the feature flags and upgrade rabbitmq-server to 3.11 again. 

   if ! [ $(grep 'cb.asc' /etc/yum.repos.d/CarbonBlack.repo) ]; then sed -i '/gpgkey/a gpgkey=https://yum.distro.carbonblack.io/enterprise/keys/cb.asc' /etc/yum.repos.d/CarbonBlack.repo; fi && rpm -e --nodeps rabbitmq-server && sed -i -E 's/(^baseurl.*enterprise\/).*(\/\$releasever\/.*)/\17.8.0-1\2/g' /etc/yum.repos.d/CarbonBlack.repo && yum clean all && yum install --downloadonly --downloaddir=/tmp rabbitmq-server* -y --disablerepo=* --enablerepo=CarbonBlack && rpm -i /tmp/rabbitmq-server-3.10.20-1.el8.noarch.rpm && /usr/share/cb/cbservice cb-rabbitmq start && /usr/share/cb/cbrabbitmqctl enable_feature_flag all && /usr/share/cb/cbservice cb-rabbitmq stop && rpm -e --nodeps rabbitmq-server && sed -i -E 's/(^baseurl.*enterprise\/).*(\/\$releasever\/.*)/\1stable\2/g' /etc/yum.repos.d/CarbonBlack.repo && yum clean all && rm -f /tmp/rabbitmq-server-3.10.20-1.el8.noarch.rpm && yum install --downloadonly --downloaddir=/tmp rabbitmq-server* -y --disablerepo=* --enablerepo=CarbonBlack && rpm -i /tmp/rabbitmq-server-3.11.24-1.el8.noarch.rpm && /usr/share/cb/cbcheck selinux -a

2. Attempt to start services

 

Explanation of the steps or to run these individually. 

1. Adds the cb.asc public key for gpg check if its missing. 

   if ! [ $(grep 'cb.asc' /etc/yum.repos.d/CarbonBlack.repo) ]; then sed -i '/gpgkey/a gpgkey=https://yum.distro.carbonblack.io/enterprise/keys/cb.asc' /etc/yum.repos.d/CarbonBlack.repo; fi

2. Uninstall the 3.11 rabbitmq-server without dependencies 

   rpm -e --nodeps rabbitmq-server

3. Update the CarbonBlack.repo baseurl to point to 7.8.0-1 repo 

   sed -i -E 's/(^baseurl.*enterprise\/).*(\/\$releasever\/.*)/\17.8.0-1\2/g' /etc/yum.repos.d/CarbonBlack.repo

4. Clean yum cache 

   yum clean all

5. Download the rabbitmq-server-3.10.20-1 rpm package. 

   yum install --downloadonly --downloaddir=/tmp rabbitmq-server* -y --disablerepo=* --enablerepo=CarbonBlack

6. Install the 3.10.20 rabbitmq-server 

   rpm -i /tmp/rabbitmq-server-3.10.20-1.el8.noarch.rpm

7. Startup rabbitmq alone through the cbservice script 

   /usr/share/cb/cbservice cb-rabbitmq start

8. Enable the feature flags 

   /usr/share/cb/cbrabbitmqctl enable_feature_flag all

9. Stop rabbitmq alone through the cbservice script 

   /usr/share/cb/cbservice cb-rabbitmq stop

10. Uninstall the 3.10.20 rabbitmq-server without dependencies 

    rpm -e --nodeps rabbitmq-server

11. Update the CarbonBlack.repo to point back to the stable (latest) release. 

    sed -i -E 's/(^baseurl.*enterprise\/).*(\/\$releasever\/.*)/\1stable\2/g' /etc/yum.repos.d/CarbonBlack.repo

12. Clean the yum cache 

    yum clean all

13. Remove the 3.10.20 download from /tmp. 

    rm -f /tmp/rabbitmq-server-3.10.20-1.el8.noarch.rpm

14. Download the 3.11.24 rabbitmq-server 

    yum install --downloadonly --downloaddir=/tmp rabbitmq-server* -y --disablerepo=* --enablerepo=CarbonBlack

15. Install the 3.11.24 rabbitmq-server rpm 

    rpm -i /tmp/rabbitmq-server-3.11.24-1.el8.noarch.rpm

16. If selinux is enabled, the permissions labels may need to be updated. 

    /usr/share/cb/cbcheck selinux -a

17. Attempt to start services. 

Additional Information

https://www.rabbitmq.com/blog/2022/07/20/required-feature-flags-in-rabbitmq-3.11