EDR: Services Will Not Start Due to RabbitMQ Timeout

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Startup fails after starting the cb-rabbitmq service.
• /var/log/messages : 

Traceback (most recent call last):
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/exceptions.py", line 84, in decorator
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/main.py", line 172, in main
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/main.py", line 112, in run
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 162, in execute
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 183, in _wait_for_network_connection
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 143, in execute
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/pyrabbit2/api.py", line 259, in is_alive
resp = self._call(uri, 'GET')
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/pyrabbit2/api.py", line 123, in _call
resp = self.http.do_call(path, method, body, headers, params)
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/pyrabbit2/http.py", line 98, in do_call
raise NetworkError("Timeout while trying to connect to RabbitMQ")
pyrabbit2.http.NetworkError: Timeout while trying to connect to RabbitMQ

• /var/log/cb/datastore/debug.log:

2021-01-27 15:38:53,809 - [WARN] - from com.rabbitmq.client.impl.ForgivingExceptionHandler in AMQP Connection 127.0.0.1:5004
An unexpected connection driver error occured (Exception message: Socket closed)

2021-01-27 15:38:53,814 - [ERROR] - from com.carbonblack.cbfs.http.listeners.WebAppContextListener in main
Initialization error, exiting
java.util.concurrent.TimeoutException: null
        at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:77)
        at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:120)
        at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:494)
        at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:315)
        at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
        at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:134)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:997)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:956)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:914)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1099)
        at com.carbonblack.core.notifications.rabbitmq.RabbitMQConnection.<init>(RabbitMQConnection.java:53)
        .......

Environment

• EDR Server: All Supported Versions

Cause

• DNS is likely misconfigured and Rabbitmq is timing out before it can be resolved.
  • The cb-enterprise service will attempt to resolve the addresses using each of the entries in the resolv.conf file before moving on to the next step.  If any of the DNS servers are slow to respond, the CB service can timeout before the full response is received from DNS.  We've seen the first entry be incorrect and the service timeout before it can hit the second entry with a DNS attempt.

Resolution

• Ensure the proper DNS addresses are listed in /etc/resolv.conf
• Stop all services that may still be running, individually.
• Attempt to  start cb-enterprise service.