EDR: Aggregate sensor event queue is growing too large

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• WebUI console Server Dashboard shows Aggregate sensor event queue continuing to grow excessively larger.
• Seeing these JVM GC messages orver 20% in /var/log/messages of one or moreEDR nodes

Feb 12 14:57:25 prn-cb06 cb-enterprised[4139]: cb.enterprise.tasks.server_health_monitor.indicators.garbage_collection - JVM GC for cb-datastore is at 203.7%. Current threshold is 20.0% 
Feb 12 14:58:27 prn-cb06 cb-enterprised[4139]: cb.enterprise.tasks.server_health_monitor.indicators.garbage_collection - JVM GC for cb-datastore is at 143.3%. Current threshold is 20.0%

• /var/log/cb/datastore/debug.log shows these WARNings:

2019-02-12 14:56:01,819 - [WARN] - from org.eclipse.jetty.http.HttpParser in qtp2059904228-932 
badMessage: java.lang.IllegalStateException: too much data after closed for 
HttpChannelOverHttp@6b8eebb2{r=1,c=false,a=IDLE,uri=-}

 

• /var/log/cb/access.log showing excessive "503" errors:

cat /var/log/cb/nginx/access.log | cut -d'"' -f3 | cut -d' ' -f2 | sort | uniq -c 

1406550 200 
1 204 
18 400 
39 402 
48 403 
827 408 
273 499 
12224 502 
232598 503       << excessive 503 errors
73882 504

Environment

• EDR Server: All versions
• EDR Sensors: All Versions

Cause

SOLR may not have enough allocated memory. Try increasing JAVA memory like so on all the nodes:

Resolution

1. For EDR Server versions 7.x see this article on how to allocate more RAM to both Datastore and SOLR JVM's: https://community.carbonblack.com/t5/Knowledge-Base/EDR-Sensor-backlog-growing-with-many-503s-in-Nginx/ta-p/68426  increase available Java memory on all nodes:. 
2. For earlier 6.0-6.2.x EDR servers, increase available Java memory on all nodes: 
   1. Edit /etc/cb/solr5/solr.in.sh
   2. Change 0.40 to 0.60 as so: 

XMAX=`grep MemTotal /proc/meminfo | awk '{printf("%dM", 0.40*$2/1024)}'`

To 

XMAX=`grep MemTotal /proc/meminfo | awk '{printf("%dM", 0.60*$2/1024)}'`

2. Then restart services:

/usr/share/cb/cbluster stop 
/usr/share/cb/cbcluster start

Additional Information

Future upgrades of the server Server from 6.2.x will get an rpmnew file for the solr.in.sh.