How to Use CB Qualifier to Verify Storage Drive Speeds for Existing EDR Servers

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

How to run the CB Qualifier Utility

• To estimate the storage drives ability to handle the known load from sensors on an existing EDR server. 

Environment

• EDR Server: All Supported Versions
• RHEL/CentOS: All Supported Versions

Resolution

Pre-requisites

1. Verify the storage volume being used for data storage.

   grep 'DatastoreRootDir' /etc/cb/cb.conf

2. Verify the volume has at least 100GB of free space available.

   du -h

    
3. Services will need to be down to run this test properly, a large instance could take a few hours. Schedule accordingly.
4. If this is a Clustered instance, the test will need to be run on just one eventful minion node. 

 

Determining Multiplication Factor

If support has provided a multiplication factor during troubleshooting, this can be skipped. 

1. Log into the EDR console.
2. Navigate to the "Server Dashboard" icon in the left panel.
3. Expand out the "Sharding" for each node.
4. View the "Document Count" and the core dates.
5. Calculate the average documents per day for each core. By using the dates, divide the document count by how many days this core was active.
6. In the "Server Dashboard" page, view the Sensor Statistics section and "Aggregate Sensor Event Queue".
   • This is the overall backlog of all active sensors. Backlog will never be zero. 
   • Calculate the estimated document count not ingested, using 3.6KB as an average document size. 

     Backlog in GB
     (backlog / 1048576) / 3.6
     
     Backlog in MB 
     (backlog / 1024) / 3.6

7. Use the higher amount of average documents per day from your calculations in step 5. If there is backlog it's good practice to add 20% of the backlog calculation to average document count per day. Age of backlogged events cannot be determined.

 

Running the Storage Drive Test

1. Stop the services.

   Standalone:
   /usr/share/cb/cbservice cb-enterprise stop
   
   Cluster:
   /usr/share/cb/cbcluster stop

2. Using the multiplication factor calculated or provided by support, run this command. The average document count per day in millions.

   cbr-qualifier disk -p million_process_doc -m <average_docs_per_day_in_millions> -o edr_qual.yml <DatastoreRootDir>

   Example of 338 million documents per day with DatastoreRootDir of /var/cb/data:

   cbr-qualifier disk -p million_process_doc -m 338 -o edr_qual.yml /var/cb/data

3. After the test has completed, start services again. 
4. Reading the output. Reading the CB Qualifier Storage Drive Utility Results

Additional Information