EDR: Sensor backlog growing with many 503s in Nginx
book
Article ID: 285870
calendar_today
Updated On:
Products
Carbon Black EDR (formerly Cb Response)
Issue/Introduction
Nginx shows more 503 responses for sensor communication than 200 responses
Slow response with queries in Console
/var/log/cb/enterprise/enterprise.log shows Datastore garbage collection extremely over threshold repeatedly
JVM GC for cb-datastore is at ###%. Current threshold is 20.0%
Environment
EDR Server: All Versions
Cause
Datastore is overwhelmed by the amount of incoming data and this causes increased memory usage. An increase in memory usage often leads to more Java garbage collection, which can have the side effect of slowing sensor event ingestion.
Resolution
Increase the allotted memory for Datastore in /etc/cb/cb.conf
Versions 6.2.2 and earlier:
Change DatastoreJvmMax=10% to DatastoreJvmMax=20%
Versions 6.2.3 and higher:
Add following content to the /etc/cb/cb.conf file:
# ============================== # SECTION - CbAllocate Options # ============================== #
# Enables the dynamic memory allocation for Solr Jvm services
EnableDynamicJvmMemoryAllocation = False
# The maximum % of memory allowed to solr (Used when dynamic calculation is off)
MaximumSolrMemoryPercent = 40
# The minimum % of memory allowed to solr (Used when dynamic calculation is off)
MinimumSolrMemoryPercent = 10
# The maximum % of memory allowed to datastore (Used when dynamic calculation is on or off)
MaximumDatastoreMemoryPercent = 20
# The minimum % of memory allowed to datastore (Used when dynamic calculation is on or off)
MinimumDatastoreMemoryPercent = 5