Services Will Not Start Due to RabbitMQ Timeout

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Startup fails after starting the cb-rabbitmq service.
• /var/log/messages : 

Traceback (most recent call last):
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/exceptions.py", line 84, in decorator
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/main.py", line 172, in main
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/main.py", line 112, in run
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 162, in execute
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 183, in _wait_for_network_connection
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 143, in execute
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/pyrabbit2/api.py", line 259, in is_alive
resp = self._call(uri, 'GET')
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/pyrabbit2/api.py", line 123, in _call
resp = self.http.do_call(path, method, body, headers, params)
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/pyrabbit2/http.py", line 98, in do_call
raise NetworkError("Timeout while trying to connect to RabbitMQ")
pyrabbit2.http.NetworkError: Timeout while trying to connect to RabbitMQ

• /var/log/cb/datastore/debug.log:

2021-01-27 15:38:53,809 - [WARN] - from com.rabbitmq.client.impl.ForgivingExceptionHandler in AMQP Connection 127.0.0.1:5004
An unexpected connection driver error occured (Exception message: Socket closed)

2021-01-27 15:38:53,814 - [ERROR] - from com.carbonblack.cbfs.http.listeners.WebAppContextListener in main
Initialization error, exiting
java.util.concurrent.TimeoutException: null
        at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:77)
        at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:120)
        at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:494)
        at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:315)
        at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
        at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:134)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:997)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:956)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:914)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1099)
        at com.carbonblack.core.notifications.rabbitmq.RabbitMQConnection.<init>(RabbitMQConnection.java:53)
        .......

Environment

• Carbon EDR Server: All Supported Versions

Cause

DNS is likely misconfigured and Rabbitmq is timing out before it can be resolved.

Resolution

• Ensure the proper DNS addresses are listed in /etc/resolv.conf
• Stop the services. How to Start, Stop and Restart EDR Application Services
• Reset the failure (on each node for clusters).

  systemctl reset-failed

• Start thes servivces again. How to Start, Stop and Restart EDR Application Services

Additional Information

• The cb-enterprise service will attempt to resolve the addresses using each of the entries in the resolv.conf file before moving on to the next step. 
• If any of the DNS servers are slow to respond, the CB service can timeout before the full response is received from DNS.  We've seen the first entry be incorrect and the service timeout before it can hit the second entry with a DNS attempt.