CB Response: Why there is difference in Total Number of Results on Process Search Page

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Difference in CBR Total Number of Results on Process Search Page

Example:

• If we perform an empty process search and it returns the total number of results as 2,000,000 that means All documents: 2,000,000
• Now if we search "filemod:test.txt" and the total number of results return 30
• Now if we search "-filemod:text.txt" documents without filemod "text.txt", it should return 2,000,000 - 30, which is equal to 1,999,970. However the result will be different like 1,999,990

Why there is difference in the number of results.
 

Environment

• CB Response Server: All Supported Versions

Resolution

These results are expected. The way solr stores documents is that not every field is present in every document. If we have a segment of a process stored in solr and that segment did not do any filemods, then the filemod and filemod_complete fields will not be part of that document. When you go to search for a document that does not have a particular filemod as in the example, you will get all documents that do not have the filemod field.

We can only remove results from documents where those fields do exist. For example:

All documents: 2,000,000
Docs with filemod:test.txt: 30
Docs without filemod:test.txt: 1,999,900
So that means there are 70 documents without any filemods, not just without filemods of test.txt.

This can happen for almost all our fields as well, including things like process_name.

Where relational databases like SQL have every column represented in every row, these negations make sense. However with Solr, not every field (or column) is in every stored document. Negations can become confusing in this case. (but are still accurate)