Rabbitmq failed to start on Carbon Black EDR on-prem server due to "TimeoutError"

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• EDR Server cb-enterprise service does not start
• /var/log/messages shows

cb-enterprise[359139]: TimeoutError: timed out
cb-enterprise[359139]: The above exception was the direct cause of the following exception:
cb-enterprise[359139]: Traceback (most recent call last):
cb-enterprise[359139]:  File "/usr/share/cb/virtualenv/lib/python3.10/site-packages/cb/utils/exceptions.py", line 131, in decorator
cb-enterprise[359139]:  File "/usr/share/cb/virtualenv/lib/python3.10/site-packages/cb/maintenance/cbstartup/main.py", line 178, in main
cb-enterprise[359139]:  File "/usr/share/cb/virtualenv/lib/python3.10/site-packages/cb/maintenance/cbstartup/main.py", line 118, in run
cb-enterprise[359139]:  File "/usr/share/cb/virtualenv/lib/python3.10/site-packages/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 155, in execute

• /var/log/cb/supervisord/supervisord.log shows Rabbitmq stopping right after startup:

2025-07-24 09:39:36,797 INFO success: cb-datagrid entered RUNNING state, process has stayed up for > than 8 seconds (startsecs)
2025-07-24 09:40:12,888 INFO spawned: 'cb-redis' with pid 358833
2025-07-24 09:40:20,898 INFO success: cb-redis entered RUNNING state, process has stayed up for > than 8 seconds (startsecs)
2025-07-24 09:40:21,345 INFO spawned: 'cb-rabbitmq' with pid 358865
2025-07-24 09:40:29,441 INFO success: cb-rabbitmq entered RUNNING state, process has stayed up for > than 8 seconds (startsecs)
2025-07-24 09:41:55,350 INFO waiting for cb-rabbitmq to stop

• /var/log/cb/datagrid/debug.log shows datagrid stopping right after startup:

  [WARN] - from com.carbonblack.datagrid.persist.DbPool in main
  Error testing SQL connection: java.sql.SQLException: Cannot create PoolableConnectionFactory (The connection attempt failed.)

• /var/log/cb/datastore/debug.log shows rabbitmq connection error:

Instance creation error : could not create instance for [Singleton:'com.carbonblack.cbfs.storage_model_4.DataStore$Engine',binds:com.carbonblack.cbfs.DataStoreAPI$DataStore]: java.net.ConnectException: Connection refused (Connection refused)
	java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
	java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
	java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
	java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	java.base/java.net.Socket.connect(Socket.java:609)
	com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:61)

Environment

• EDR Server: All Supported Versions
• CentOS/RHEL 7/8

Cause

• Unreachable DNS configured in /etc/resolv.conf
• hostname not correctly configured in /etc/hosts
• No internet access | Air-gaped system

Resolution

• Add correct hostname to /etc/hosts file
• To check hostname of the node, run

Example:

[root@myhostname] $ hostname
myhostname

• Check the hostname is configured in the /etc/hosts file. If not, add the hostname to the file.

Example:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1	myhostname
# {{BEGIN - CB Server}}
# The block within these BEGIN/END tags is managed by Carbon Black Enterprise
# Server configuration tools. DO NOT EDIT within BEGIN/END comment tags as the
# content of this section may get automatically regenerated.

127.0.0.1           CB-SERVER-CLUSTER-HEAD-NODE

# {{END - CB Server}}

 

• Check how many DNS are configured and are they reachable. 

Example:

[root@localhost # nslookup
> server
Default server: ##.##.##.#
Address: ##.##.##.##
Default server: #.#.#.#
Address: #.#.#.#

cat /etc/resolv.conf
# Generated by NetworkManager
search localdomain
nameserver ##.##.##.#
nameserver #.#.#.#

 

• Remove the unreachable DNS from /etc/resolv.conf
• If this is a new server install, and DNS entries were problematic when running cbinit, you should rerun cbinit
• Restart CB Response services

            


 

Additional Information

• Removing #.#.#.# (not reachable) on an air-gapped system, resolved rabbitmq stuck issue and services resumed.
• Checking source of unreachable DNS will need to be traced, or problem may reoccur after network manager restart.
• Check DHCP if configured.