RabbitMQ Fails to Start After a Server Upgrade on an Airgapped System

Products

• Carbon Black EDR

Issue/Introduction

When upgrading to 7.8.1+, RabbitMQ fails to start or times out.

• Running cb-enterprise status shows cb-rabbitmq as either:

cb-rabbitmq                                                               STARTING
cb-rabbitmq           Exited too quickly (process log may have details)   BACKOFF

• In /var/log/cb/rabbitmq/startup_log there is a message similar to this: 

Starting rabbitmq
2025-11-20 09:52:06.303110-06:00 [error] <0.234.0> Feature flags: `stream_single_active_consumer`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
2025-11-20 09:52:06.309796-06:00 [error] <0.234.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
2025-11-20 09:52:06.309796-06:00 [error] <0.234.0>                                               stream_single_active_consumer}
2025-11-20 09:52:06.316446-06:00 [error] <0.234.0>
2025-11-20 09:52:06.316446-06:00 [error] <0.234.0> BOOT FAILED
2025-11-20 09:52:06.316446-06:00 [error] <0.234.0> ===========
2025-11-20 09:52:06.316446-06:00 [error] <0.234.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}

Environment

EDR Server 7.8.1+

Cause

• The required steps from the release notes were not followed.
• The previous EDR server did not have the RabbitMQ feature flags enabled required for rabbitmq-server 3.11+

Resolution

I. Download the older rabbitmq-server RPM's from the caching server

1. Authenticate to the Packages Repository
2. Update the CarbonBlack.repo baseurl to point to 7.8.0-1 repository
   • vi /etc/yum.repos.d/CarbonBlack.repo
3. Change the last portion of the baseurl from 7.9.0-1 to 7.8.0-1
   • ...packages.broadcom.com/artifactory/carbonblack-edr/7.8.0-1/$releasever/$basearch
4. Clean the yum cache
   • yum clean all
5. Download the rabbitmq-server-3.10.20-1 rpm package.
   • yum install --downloadonly --downloaddir=/tmp/3.10.20 rabbitmq-server* -y --disablerepo=* --enablerepo=CarbonBlack
6. Update the CarbonBlack.repo baseurl to point to 7.8.1 repository
   • vi /etc/yum.repos.d/CarbonBlack.repo
7. Change the last portion of the baseurl from 7.8.0-1 to 7.8.1
   • ...packages.broadcom.com/artifactory/carbonblack-edr/7.8.1/$releasever/$basearch
8. Download the rabbitmq-server-3.11.24 rpm package.
   • yum install --downloadonly --downloaddir=/tmp/3.11.24 rabbitmq-server* -y --disablerepo=* --enablerepo=CarbonBlack
9. Change the last portion of the baseurl from 7.8.1 to 7.9.0-1
   • ...packages.broadcom.com/artifactory/carbonblack-edr/7.9.0-1/$releasever/$basearch
10. Clean the yum cache
    • yum clean all
11. Download the rabbitmq-server-3.12.13-1 rpm package.
    • yum install --downloadonly --downloaddir=/tmp/3.12.13 rabbitmq-server* -y --disablerepo=* --enablerepo=CarbonBlack
12. Continue with downloading the rabbitmq-server RPM's for any newer server version following the same logic
13. Zip the RPMs and move them to the air-gapped server to the /tmp folder

II. Downgrading RabbitMQ and enabling all feature flags

1. Unzip the file and move the 3 RPMs to the air-gapped server to the /tmp folder
2. Stop services:
   • /usr/share/cb/cbservices cb-enterprise stop
3. Uninstall the 3.11 rabbitmq-server without dependencies.
   • rpm -e --nodeps rabbitmq-server
4. Install the 3.10.20 rabbitmq-server
   • rpm -i --nodeps /tmp/rabbitmq-server-3.10.20-1.el8.noarch.rpm
5. Start the rabbitmq alone through the cbservice script
   • /usr/share/cb/cbservice cb-rabbitmq start
6. Enable the feature flags
   • /usr/share/cb/cbrabbitmqctl enable_feature_flag all
7. Stop rabbitmq alone through the cbservice script
   • /usr/share/cb/cbservice cb-rabbitmq stop
8. Uninstall the 3.10.20 rabbitmq-server without dependencies
   • rpm -e --nodeps rabbitmq-server
9. Install the 3.11.24 rabbitmq-server
   • rpm -i --nodeps /tmp/rabbitmq-server-3.11.24-1.el8.noarch.rpm
10. Start the rabbitmq alone through the cbservice script
    • /usr/share/cb/cbservice cb-rabbitmq start
11. Enable the feature flags
    • /usr/share/cb/cbrabbitmqctl enable_feature_flag all
12. Stop rabbitmq alone through the cbservice script
    • /usr/share/cb/cbservice cb-rabbitmq stop
13. Uninstall the 3.11.24 rabbitmq-server without dependencies
    • rpm -e --nodeps rabbitmq-server
14. Install the 3.12.13 rabbitmq-server
    • rpm -i --nodeps /tmp/rabbitmq-server-3.12.13-1.el8.noarch.rpm
15. Continue downgrading rabbitmq-server for any newer server versions following the same logic
16. If selinux is enabled, the permissions labels may need to be updated.
    • /usr/share/cb/cbcheck selinux -a
17. Start the services.
    • /usr/share/cb/cbservices cb-enterprise start
18. Check the status of RabbitMQ
    • /usr/share/cb/cbservices cb-enterprise status

Additional Information

https://www.rabbitmq.com/blog/2022/07/20/required-feature-flags-in-rabbitmq-3.11