EDR Server: Watchlists and feeds are no longer running

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• The process watchlist_search has been running for several hours or days
• The process watchlist_search is utilizing a high amount of CPU
• /var/log/cb/job-runner/job-runner.log shows the following every time watchlist_search runs

  • Unable to grab exclusive mutex for watchlist searcher, another watchlist searcher may be running

• job-runner.log shows a watchlist timeout when search started to fail

• /var/log/cb/solr/debug.log shows the watchlist still running
• Core optimizataion fails while the watchlists run

Environment

• EDR (formerly CB Response) Server: 6.2.3 and lower

Cause

• Nested queries running longer, most noticeably with modload searches - CB-14781.
• Watchlist queries returning incomplete results lock up watchlist searches for hours - CB-17415.

 

Resolution

• Upgrade to 6.3 or higher to resolve both CB-17415 and CB-14781
• Upgrade to 6.2.3 to resolve CB-17415
• Workaround if an upgrade is not possible

1. Determine which watchlist or threat report query is causing the hang
2. Disable the query in the UI

Additional Information

The watchlist_search will eventually stop, but can be stopped maually