EDR: One or more Server Nodes fail to start because stuck process remains running

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• One or more secondary server nodes fails to start when a cluster start command is issued.
• While the cluster is stopped, one or more unexpected Carbon Black processes remain running.

Environment

• EDR: 6.x and Higher
• Clustered environment

Cause

Some EDR processes may not shut down properly when the cluster is stopped. Their presence can prevent the cluster from restarting properly in the future. 

Resolution

1. From the primary server, stop the EDR cluster:

# /usr/share/cb/cbcluster stop

2. Check for stuck processes:

# ps -ef | grep cb

Example output:

cb  12345 10011  0 1 03:10 /usr/lib64/erlang/erts-9.3/bin/epmd -daemon
myusername  19846 19718  0 17:15 pts/1    00:00:00 ps -ef
myusername  19847 19718  0 17:15 pts/1    00:00:00 grep cb

3. Stop the stuck process (PID 12345 in this example)

# kill 12345

4. Verify the process was removed:

# ps -ef | grep cb

5. Restart the cluster from the master server:

# /usr/share/cb/cbcluster start

 

Additional Information

• The /usr/lib64/erlang/erts-9.3/bin/epmd process is only an example
• Other processes may be stuck