EDR: cb-enterprise Services Fails to Start on Master Node

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• 'systemctl start cb-enterprise' cmd failed to start all the services
• services start appears stuck on 'cb-datagrid' service 
  • Example: Waiting for cb-datagrid to initialize.................

• 'journalctl -fexu cb-enterprise' output shows the following:

  -- Unit cb-enterprise.service has begun starting up.
  Sep 19 09:48:19 <servername> cb-enterprise[19954]: Redirecting to /bin/systemctl status crond.service
  Sep 19 09:48:19 <servername> cb-enterprise[19954]: Redirecting to /bin/systemctl status rsyslog.service
  Sep 19 09:48:22 <servername> runuser[20080]: pam_unix(runuser:session): session opened for user cb by (uid=0)
  Sep 19 09:48:23 <servername> runuser[20080]: pam_unix(runuser:session): session closed for user cb
  Sep 19 09:48:25 <servername> cb-enterprise[19954]: Starting cb-supervisord (via systemctl): [ OK ]
  Sep 19 09:48:26 <servername> cb-enterprise[19954]: Starting cb-pgsql: [ OK ]
  Sep 19 09:48:28 <servername> cb-enterprise[19954]: Starting cb-datagrid: [ OK ]
  Sep 19 09:51:12 <servername> cb-enterprise[19954]: Waiting for cb-datagrid to initialize.................
  Sep 19 09:51:12 <servername> systemd[1]: cb-enterprise.service: control process exited, code=exited status=1
  Sep 19 09:51:12 <servername> systemd[1]: Failed to start SYSV: Carbon Black is a surveillance camera for your computer -- always recording so you know precisely what happened and where. This component provides an internal interface to the primary datastore..
  -- Subject: Unit cb-enterprise.service has failed
  -- Defined-By: systemd
  -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
  --
  -- Unit cb-enterprise.service has failed.

Environment

• EDR Server: All Versions

Cause

Another service not related to EDR is using a necessary port for one or more of the services.
Hanging zombie processes

Resolution

1. Confirm all processes by user cb are no longer running by using the command below:

   ps -aef | grep cb
   

2. If any exist, kill all running by the user cb
3. From the terminal, run "netstat -nltp" and find the PID for any service running on an EDR port that is not owned by the user cb. What Ports are used by Server Services
4. Kill the PID of the service to release the port
5. For clusters, RabbitMQ should have the mnesia directories cleared. How to reset Mnesia for RabbitMQ

kill -9 <PID>

3. Restart the services: Restarting services