EDR: RabbitMQ failing to start with invalid credentials

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

RabbitMQ failing to start with invalid credentials
Possible error messages:

2019-05-07 13:27:13,878 - [WARN] - from com.rabbitmq.client.impl.ForgivingExceptionHandler in AMQP Connection 127.0.0.1:5004 
An unexpected connection driver error occured (Exception message: Connection reset) 

2019-05-07 13:27:13,880 - [ERROR] - from com.carbonblack.cbfs.http.listeners.WebAppContextListener in main 
Initialization error, exiting 
com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.

 

=ERROR REPORT==== 7-May-2019::09:37:19 === 
Error on AMQP connection <0.18442.21> (127.0.0.1:39864 -> 127.0.0.1:5004, state: starting): 
AMQPLAIN login refused: user 'cb' - invalid credentials

Environment

• EDR (Carbon Black Response) Server: All Versions

Cause

User cb is missing or password is incorrect

Resolution

 

1. Log into the back-end server via SSH/Terminal
2. Start cb-rabbitmq if services have been stopped

   /usr/share/cb/cbservice cb-rabbitmq start

3. Grep for the RabbitMQ password
   1. 7.7.0+ : Please open a support ticket with this token output, support can convert and provide you the password. 

      grep 'RabbitMQToken' /etc/cb/cb.conf

   2. 7.6.2 and below

      grep 'RabbitMQPassword' /etc/cb/cb.conf

4. Copy the output to be entered later
5. Check to see if the cb user exists, perform on primary and minions. 

   /usr/share/cb/cbrabbitmqctl list_users 

6. Utilize the /usr/share/cb/cbrabbitmqctl to update or add the user, perform on primary and minions
   • If the cb user is returned 

     /usr/share/cb/cbrabbitmqctl change_password cb <password> 
     /usr/share/cb/cbrabbitmqctl set_permissions -p / cb ".*" ".*" ".*"

   • if user is missing: 

     /usr/share/cb/cbrabbitmqctl add_user cb <password> 
     /usr/share/cb/cbrabbitmqctl set_user_tags cb administrator 
     /usr/share/cb/cbrabbitmqctl set_permissions -p / cb ".*" ".*" ".*"

7. Stop all services

   Standalone: 
   /usr/share/cb/cbservice cb-enterprise stop
   
   Cluster:
   /usr/share/cb/cbcluster stop

8. On primary and minions, clean out the old RabbitMQ leftovers if they remain

   rm -rf /var/cb/data/rabbitmq/mnesia
   rm -f /var/cb/.erlang.cookie

9. Start the services again

   Standalone: 
   /usr/share/cb/cbservice cb-enterprise start
   
   Cluster:
   /usr/share/cb/cbcluster start

Additional Information

• cb user may be missing after a disk full situation and mnesia directory was cleared to restart
• To check user credentials are set correct

  • /usr/share/cb/cbrabbitmqctl authenticate_user cb <password>

• rabbitmq must be running for perform steps 4 and 5

  • service cb-rabbitmq start