EDR: Rabbitmq failed to start
search cancel

EDR: Rabbitmq failed to start

book

Article ID: 288478

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

  • CB Response Services does not start.
  • /var/log/messages shows
Jul 20 13:29:34 dscb001 cb-enterprise[1114]: raise NetworkError("Timeout while trying to connect to RabbitMQ")
Jul 20 13:29:34 dscb001 cb-enterprise[1114]: pyrabbit2.http.NetworkError: Timeout while trying to connect to RabbitMQ
Jul 20 13:29:34 dscb001 systemd[1]: cb-enterprise.service: control process exited, code=exited status=1
 
  • /var/log/cb/solr/debug.log shows:
2020-07-21 14:33:16,491 - [ERROR] - from com.carbonblack.cbfs.solr.CbEventPublisher in qtp1529306539-20 
RabbitMQConnection failed, event notifications will be disabled
java.util.concurrent.TimeoutException: null

2020-07-21 14:33:21,593 - [WARN] - from com.rabbitmq.client.impl.ForgivingExceptionHandler in AMQP Connection 127.0.0.1:5004 
An unexpected connection driver error occured (Exception message: Socket closed)


 

Environment

  • EDR Server: 6.x and Higher
  • CentOS: 7.8 

Cause

  • Unreachable DNS configured in /etc/resolv.conf
  • hostname not correctly configured in /etc/hosts
  • No internet access | Air-gaped system

Resolution

  • Add correct hostname to /etc/hosts file
  • To check hostname of the node, run
Example:

[root@myhostname] $ hostname
myhostname
  • Check the hostname is configured in the /etc/hosts file. If not, add the hostname to the file.
Example:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1	myhostname
# {{BEGIN - CB Server}}
# The block within these BEGIN/END tags is managed by Carbon Black Enterprise
# Server configuration tools. DO NOT EDIT within BEGIN/END comment tags as the
# content of this section may get automatically regenerated.

127.0.0.1           CB-SERVER-CLUSTER-HEAD-NODE

# {{END - CB Server}}
 
  • Check how many DNS are configured and are they reachable. 
Example:

[root@localhost cbevents_2020_07_20_0555]# nslookup
> server
Default server: 10.40.40.1
Address: 110.40.40.1#53
Default server: 8.8.8.8
Address: 8.8.8.8#53

cat /etc/resolv.conf
# Generated by NetworkManager
search localdomain
nameserver 10.40.40.1
nameserver 8.8.8.8
              


 

Additional Information

  • Removing 8.8.8.8 (not reachable) on an air-gapped system, resolved rabbitmq stuck issue and services resumed.
  • Checking source of unreachable DNS will need to be traced, or problem may reoccur after network manager restart.
  • Check DHCP if configured.