RabbitMQ failing to start with invalid credentials

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

RabbitMQ failing to start with invalid credentials
Possible error messages:

2019-05-07 13:27:13,878 - [WARN] - from com.rabbitmq.client.impl.ForgivingExceptionHandler in AMQP Connection 127.0.0.1:5004 
An unexpected connection driver error occured (Exception message: Connection reset) 

2019-05-07 13:27:13,880 - [ERROR] - from com.carbonblack.cbfs.http.listeners.WebAppContextListener in main 
Initialization error, exiting 
com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.

=ERROR REPORT==== 7-May-2019::09:37:19 === 
Error on AMQP connection <0.18442.21> (127.0.0.1:39864 -> 127.0.0.1:5004, state: starting): 
AMQPLAIN login refused: user 'cb' - invalid credentials

Environment

Carbon Black EDR: All Versions

Cause

User cb is missing or password is incorrect

Resolution

Log into the back-end server via SSH/Terminal
On primary and minions, clean out the old RabbitMQ leftovers if they remain.
```
rm -rf /var/cb/data/rabbitmq/mnesia
rm -f /var/cb/.erlang.cookie
```
Start cb-rabbitmq if services have been stopped
```
/usr/share/cb/cbservice cb-rabbitmq start
```
If RabbitMQ fails to start, it's likely Selinux related. Run this command to set the correct labels. Then stop and start cb-rabbitmq again.
```
/usr/share/cb/cbcheck selinux -a
```
Check to see if the cb user exists, perform on primary and minions.
```
/usr/share/cb/cbrabbitmqctl list_users 
```

Utilize the /usr/share/cb/cbrabbitmqctl to update or add the user, perform on primary and minions

If the cb user is returned

/usr/share/cb/cbrabbitmqctl set_user_tags cb administrator
/usr/share/cb/cbrabbitmqctl set_permissions -p / cb ".*" ".*" ".*"

if the cb user is missing, recreate the user with a random temp password, the product will regenerate the salted password upon cb-enterprise startup:

password=$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 20); /usr/share/cb/cbrabbitmqctl add_user cb "$password"
/usr/share/cb/cbrabbitmqctl set_user_tags cb administrator 
/usr/share/cb/cbrabbitmqctl set_permissions -p / cb ".*" ".*" ".*"

Stop all services

Standalone: 
/usr/share/cb/cbservice cb-enterprise stop

Cluster:
/usr/share/cb/cbcluster stop

Start the services again

Standalone: 
/usr/share/cb/cbservice cb-enterprise start

Cluster:
/usr/share/cb/cbcluster start

Additional Information

cb user may be missing after a disk full situation and mnesia directory was cleared to restart.
7.7.0+ Salts the password based on the RabbitMQToken in /etc/cb/cb.conf. Following this process will regenerate the salted password using the token during the startup script. The steps in 6b are setting a random temporary password only to create the cb user again.