EDR: RabbitMQ failing to start with invalid credentials
search cancel

EDR: RabbitMQ failing to start with invalid credentials

book

Article ID: 285719

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

RabbitMQ failing to start with invalid credentials
Possible error messages:
2019-05-07 13:27:13,878 - [WARN] - from com.rabbitmq.client.impl.ForgivingExceptionHandler in AMQP Connection 127.0.0.1:5004 
An unexpected connection driver error occured (Exception message: Connection reset) 

2019-05-07 13:27:13,880 - [ERROR] - from com.carbonblack.cbfs.http.listeners.WebAppContextListener in main 
Initialization error, exiting 
com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
 
=ERROR REPORT==== 7-May-2019::09:37:19 === 
Error on AMQP connection <0.18442.21> (127.0.0.1:39864 -> 127.0.0.1:5004, state: starting): 
AMQPLAIN login refused: user 'cb' - invalid credentials

Environment

  • EDR (Carbon Black Response) Server: All Versions

Cause

User cb is missing or password is incorrect

Resolution

 
  1. Log into the back-end server via SSH/Terminal
  2. Start cb-rabbitmq if services have been stopped
    /usr/share/cb/cbservice cb-rabbitmq start
  3. Grep for the RabbitMQ password
    1. 7.7.0+ : Please open a support ticket with this token output, support can convert and provide you the password. 
      grep 'RabbitMQToken' /etc/cb/cb.conf
    2. 7.6.2 and below
      grep 'RabbitMQPassword' /etc/cb/cb.conf
  4. Copy the output to be entered later
  5. Check to see if the cb user exists, perform on primary and minions. 
    /usr/share/cb/cbrabbitmqctl list_users 
  6. Utilize the /usr/share/cb/cbrabbitmqctl to update or add the user, perform on primary and minions
    • If the cb user is returned 
      /usr/share/cb/cbrabbitmqctl change_password cb <password> 
      /usr/share/cb/cbrabbitmqctl set_permissions -p / cb ".*" ".*" ".*"
    • if user is missing: 
      /usr/share/cb/cbrabbitmqctl add_user cb <password> 
      /usr/share/cb/cbrabbitmqctl set_user_tags cb administrator 
      /usr/share/cb/cbrabbitmqctl set_permissions -p / cb ".*" ".*" ".*"
  7. Stop all services
    Standalone: 
    /usr/share/cb/cbservice cb-enterprise stop
    
    Cluster:
    /usr/share/cb/cbcluster stop
  8. On primary and minions, clean out the old RabbitMQ leftovers if they remain
    rm -rf /var/cb/data/rabbitmq/mnesia
    rm -f /var/cb/.erlang.cookie
  9. Start the services again
    Standalone: 
    /usr/share/cb/cbservice cb-enterprise start
    
    Cluster:
    /usr/share/cb/cbcluster start

Additional Information

  • cb user may be missing after a disk full situation and mnesia directory was cleared to restart
  • To check user credentials are set correct
    • /usr/share/cb/cbrabbitmqctl authenticate_user cb <password>
  • rabbitmq must be running for perform steps 4 and 5
    • service cb-rabbitmq start