Adding a Priority Queue for Sensors with the Highest Backlog

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Adding a datastore priority queue for sensors with the highest backlog. 

• Sensors that are online for a short period of time each day. 
• Help offload sensors with the highest backlog. 

Environment

• Carbon Black EDR Server: 7.6 and higher

Resolution

Important: It's suggested to open a support case with cbdiags to get configuration suggestions and path of resolution before enabling this feature. 

• Do not enable this feature for overall backlog. 
• Enabling this feature can cause backlog if too many sensors fall within the threshold.
• This feature requires available resources, a system with overall backlog should not have this configuration set. 
• Other causes such as a cloned virtual machine with events from the base image. 

 

Enabling the Priority Queue: 

1. Open /etc/cb/cb.conf on each node. 
2. Add the DatastoreEnableSensorPriority configuration to enable the feature. 

   DatastoreEnableSensorPriority=True 

3. See Additional Information for other configurations, using the default or updating based on high backlog seen on targeted sensors. 
4. Restart the services. How to Start, Stop and Restart EDR Application Services

 

Additional Information

• Once enabled, additional EDR server resources (CPU & Memory) are provided to handle the sensors high backlog.
• Once DatastoreEnableSensorPriority=True, the below default values willl be set. 

Variable	Value	Description.
DatastorePriorityBacklogThreshold	5000000	When priority is enabled, any incoming sensor which has its individual backlog more than this value would be considered for new-storage-queue
DatastorePrioritySubmitQMemoryPercent	2	Similar to DatastoreSubmitQMemoryPercent which is 10% memory by default.  However, this is the size of "new storage queue" where events are pushed before solr-writer threads pick them.
DatastorePriorityEventCoreClientThreads	0	Auto-configurable;  Determines number of "solr writer threads" against new prioirity-storage-queue that should be used for instance. [no_of_cores/8]' (min:1 max:3)