Sensor backlog growing with many 503s in Nginx

Products

• Carbon Black EDR (formerly Cb Response)
• Carbon Black EDR

Issue/Introduction

• Nginx shows more 503 responses for sensor communication than 200 responses
• Slow response with queries in Console
• /var/log/cb/enterprise/enterprise.log shows Datastore garbage collection extremely over threshold repeatedly

JVM GC for cb-datastore is at ###%. Current threshold is 20.0%

Environment

• Carbon Black EDR Server: All Versions

Cause

Datastore is overwhelmed by the amount of incoming data and this causes increased memory usage. An increase in memory usage often leads to more Java garbage collection, which can have the side effect of slowing sensor event ingestion.

Resolution

1. Increase the allotted memory for Datastore in /etc/cb/cb.conf
   • Versions 6.2.2 and earlier:
     • Change DatastoreJvmMax=10% to DatastoreJvmMax=20%
   • Versions 6.2.3 and higher:
     • Add following content to the /etc/cb/cb.conf file:

# ============================== # SECTION - CbAllocate Options # ============================== #
# Enables the dynamic memory allocation for Solr Jvm services 
EnableDynamicJvmMemoryAllocation = False 

# The maximum % of memory allowed to solr (Used when dynamic calculation is off) 
MaximumSolrMemoryPercent = 60

# The minimum % of memory allowed to solr (Used when dynamic calculation is off) 
MinimumSolrMemoryPercent = 10 

# The maximum % of memory allowed to datastore (Used when dynamic calculation is on or off) 
MaximumDatastoreMemoryPercent = 20 

# The minimum % of memory allowed to datastore (Used when dynamic calculation is on or off) 
MinimumDatastoreMemoryPercent = 5 

2. Save and exit /etc/cb/cb.conf file
3. Restart EDR services. How to Start, Stop and Restart EDR Application Services
4. Run the following to verify changes have been made in settings

   /usr/share/cb/virtualenv/bin/python -m cb.maintenance.cballocate.main -i datastore

Additional Information

• Set 'EnableDynamicJvmMemoryAllocation = True' to turn on dynamic memory allocation.
• EnableDynamicJvmMemoryAllocation attempts to calculate the memory settings for Solr and Datastore based on the total memory of the machine.
  
  • This setting attempts to reduce the over utilization of memory in smaller instances and under utilization of memory on larger instances that would occur with using a set percentage for all instances by default. 
  • Use manual only when seeing OOM and high garbage collection (GC) threshold percentages or high backlog. 
• Depending on the size of backlog, document ingestion can still take some time to go down. 
• DatastoreJvmMax has been depreciated in versions v6.2.3