Process Search or Watchlist Query Timeouts when Searches Contain Binary Metadata

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Overall slowness in the UI
• Warnings for the search appear in /var/log/cb/job-runner/job-runner.log

  • <warning> [watchlist_search] Solr request returned incomplete results after 120623 ms.

• modulestore is very large
• Solr logs show queries timing out with the message 

  • The request took too long to iterate over terms. Timeout

Environment

• Carbon Black EDR: All Versions

Cause

cbmodules core for binary metadata has grown too large to effectively return the data within a reasonable time.

Resolution

• The amount of data queried must be reduced. Consider doing one or more of the following:
  • Clear old binary data using modulestore_purge - How to Enable Automated Cbmodule Purging
  • Modify the query to be as specific as possible.
  • Remove any reference to binary fields in the query.

Additional Information

• It is recommended to be under 1.5 million binaries store in the cbmodules core.
• When doing a process search with binary metadata such as digsig_result, publisher, etc. Solr has to join between the cbevents core and cbmodules core.
  • As with any database, joins are expensive
  • Each search has to go through all binary metadata stored, the more that is stored, the longer it will take to return results. 
• Most metadata is single use binaries, for example an OS upgrade the binaries are only seen once in the environment. 
  • cbmodule_purge job allows the removal of older metadata that does not have an associated event N set days after that event has been removed from the cbevents core.