How to remove or decommission a minion that has gone offline or failed

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

There are occasions when a minion crashes or otherwise becomes unavailable. The cluster needs to be reconfigured for the now orphaned sensors of that minion to start to report into the other working minions.

Environment

• EDR Server: 6.0.x

Resolution

1. Stop the cluster from the master with the command:

   sudo /usr/share/cb/cbcluster stop

2. ​​​​​​On ALL remaining minions and the master servers, edit /etc/cb/cluster.conf
3. Modify the [Cluster] section according to the number of servers (including the master). The NodeCount value should be reduced by the number of minions being taken offline. Example: if the NodeCount is 8, and you are removing one minion, reduce the NodeCount value to 7.
4. Use hashtags # to comment out the failed minion.

[Cluster]
NodeCount=7
NextSlaveAutoInc=8 
#[Slave2]
# Host=1.2.3.4
# User=root
# HasEvents=True
# ReadOnly=False

5. Once all the cluster.conf files are complete, restart the cluster on the master:

sudo /usr/share/cb/cbcluster start

Additional Information

• For EDR clusters 7.x see the documentation here
• Once the cluster is restarted, the orphaned sensors will be re-assigned new nodes to report to. Be aware that the remaining minions that will be absorbing this additional sensor traffic will exhibit an increase the resource usage and may not be properly sized to handle the load.
• When the offline minion is restored to working order, you can revert the changes in the cluster.conf and restart the cluster. This will cause the sensors to begin checking into the previously offline minion.