Cb Response: Cluster fails to start on cb-enterprise with RabbitMQ already running

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• cb-enterprise does not start on one or more minions

  • Node[ 1 ] Error during cb-enterprise start on...

• Minions not starting show cb-rabbitmq spawning and exiting over and over

  • 2018-08-28 11:14:10,218 INFO spawned: 'cb-rabbitmq' with pid 7887
    2018-08-28 11:14:11,220 INFO success: cb-rabbitmq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
    2018-08-28 11:14:15,193 INFO exited: cb-rabbitmq (exit status 1; not expected)

• /var/log/cb/rabbitmq/startup_log has the following message

  • Starting rabbitmq
    ERROR: node with name "rabbit" already running on "CB-SERVER-CLUSTER-MINION-NODE-2

    
     

Environment

• Cb Response Cluster: All versions

Cause

RabbitMQ is still running on one or more of the minions before startup


 

Resolution

1. Stop the cluster from the master server

   • sudo /usr/share/cb/cbcluster stop

2. Run the following command on each node in the cluster to confirm RabbitMQ is stopped

   • ps -ef | grep rabbitmq

3. Kill any running processes returned from the output above, replacing <pid_of_rabbitmq> with the PID of the process

   • sudo kill -9 <pid_of_rabbitmq>

4. Once all RabbitMQ processes have been stopped, start the cluster

   • sudo /usr/share/cb/cbcluster start

Additional Information

A full list of running Cb Response services can be found via the command ps -ef | grep cb