Data Partition Disk Space on Primary Server is Exponentially Higher than the Minion

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Disk space used by the data partition is much higher on the primary than the minion. 

• Running the following command in terminal on the primary results in 0 results or just the primary servers information. 

  psql -p 5002 cb -c "select * from cluster_node_sensor_addresses;"

   
• No traffic is seen in the nginx logs from the sensors on the minion node. The following command returns 0 or low results.

  cat /var/log/cb/nginx/access.log | grep 'reserve' | wc -l

Environment

• Carbon Black EDR Server: All Versions
• Clustered

Cause

One or more entries is missing from the cluster_node_sensor_addresses table to tell the sensor which node to submit event data into. 

Resolution

1. Verify the node id's 

   cat /etc/cb/cluster.conf

   • [Primary] is always node id 0.
   • [Minion <id>] is the minions id.
2.  Run this command for each node missing the information, updating the command with the valid node id found in step 1, IP address and hostname. Note: all values should be surrounded by single quotes except the node_id and port. Do not include the <> brackets.

   psql -p 5002 cb -c "insert into cluster_node_sensor_addresses (id, address, port, ssl, hostname) values (<node_id>,'<ip addr>',443,'t','<hostname>');"

3. Stop the cluster and start it again to take affect 

   /usr/share/cb/cbcluster stop && /usr/share/cb/cbcluster start

4. Verify sensors are now submitting to the minion node. Check firewall settings if sensors are still not sending.

   cat /var/log/cb/nginx/access.log | grep 'reserve' | wc -l

 

Additional Information

• The cluster_node_sensor_addresses table is generated at first setup "cbinit" for the primary and during "cbcluster add-node" for the minion.
• With this info missing the sensor will instead use the Server URL set in the sensor groups as the datastore server and treat the cluster as a standalone instance.  
• All data already submitted to the primary will remain on the primary. Over time the cores will be deleted and the disk space will be load balanced as expected.