Process search timeouts due to Solr running low on QTP threads

Products

• Carbon Black EDR
• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Process Search, Binary Search, and Triage Alerts pages sometimes fail to load or timeout with exceptions when trying to perform queries. The Solr backend logs RejectedExecutionException errors in /var/log/cb/solr/debug.log:

2021-10-08 14:04:32,815 - [ERROR] - from org.apache.solr.servlet.HttpSolrCall in qtp319644606-1050309 [x:reader]
null:java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$150/1923810797@60a1dbd rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4d67a5c[Running, pool size = 256, active threads = 251, queued tasks = 0, completed tasks = 918178874]

Environment

EDR version 7.x

Cause

Solr uses a queued thread pool (QTP) with a default limit of 256 threads to manage queries, indexing, etc. When all the QTP threads are actively being used, any new request must wait until an active thread completes and is handed back to the pool. Methods waiting for a thread to become available will pause/block which can lead to query timeouts.

Resolution

Increase the Solr "maximumPoolSize" in /etc/cb/solr/core_conf/solr.xml.template from the default 256 threads to a larger number, such as 400.

Additional Information

The additional available QTP threads should reduce the chances of queries being blocked waiting to be assigned a thread from the pool, but it will also allow more queries to be active in memory, potentially leading to excessive heap memory usage. The maximumPoolSize might need to be reduced less than 400 to find the right balance between performance and memory usage if excessive "JVM GC" garbage collection messages are being logged in /var/log/cb/enterprise/enterprise.log.