EDR: Rabbitmq failed to start

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• CB Response Services does not start.
• /var/log/messages shows

Jul 20 13:29:34 dscb001 cb-enterprise[1114]: raise NetworkError("Timeout while trying to connect to RabbitMQ")
Jul 20 13:29:34 dscb001 cb-enterprise[1114]: pyrabbit2.http.NetworkError: Timeout while trying to connect to RabbitMQ
Jul 20 13:29:34 dscb001 systemd[1]: cb-enterprise.service: control process exited, code=exited status=1

 

• /var/log/cb/solr/debug.log shows:

2020-07-21 14:33:16,491 - [ERROR] - from com.carbonblack.cbfs.solr.CbEventPublisher in qtp1529306539-20 
RabbitMQConnection failed, event notifications will be disabled
java.util.concurrent.TimeoutException: null

2020-07-21 14:33:21,593 - [WARN] - from com.rabbitmq.client.impl.ForgivingExceptionHandler in AMQP Connection 127.0.0.1:5004 
An unexpected connection driver error occured (Exception message: Socket closed)

 

Environment

• EDR Server: 6.x and Higher
• CentOS: 7.8 

Cause

• Unreachable DNS configured in /etc/resolv.conf
• hostname not correctly configured in /etc/hosts
• No internet access | Air-gaped system

Resolution

• Add correct hostname to /etc/hosts file
• To check hostname of the node, run

Example:

[root@myhostname] $ hostname
myhostname

• Check the hostname is configured in the /etc/hosts file. If not, add the hostname to the file.

Example:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1	myhostname
# {{BEGIN - CB Server}}
# The block within these BEGIN/END tags is managed by Carbon Black Enterprise
# Server configuration tools. DO NOT EDIT within BEGIN/END comment tags as the
# content of this section may get automatically regenerated.

127.0.0.1           CB-SERVER-CLUSTER-HEAD-NODE

# {{END - CB Server}}

 

• Check how many DNS are configured and are they reachable. 

Example:

[root@localhost cbevents_2020_07_20_0555]# nslookup
> server
Default server: 10.40.40.1
Address: 110.40.40.1#53
Default server: 8.8.8.8
Address: 8.8.8.8#53

cat /etc/resolv.conf
# Generated by NetworkManager
search localdomain
nameserver 10.40.40.1
nameserver 8.8.8.8

 

• Remove the unreachable DNS from /etc/resolv.conf
• Restart CB Response services

            


 

Additional Information

• Removing 8.8.8.8 (not reachable) on an air-gapped system, resolved rabbitmq stuck issue and services resumed.
• Checking source of unreachable DNS will need to be traced, or problem may reoccur after network manager restart.
• Check DHCP if configured.