EDR: Event Ingestion Is Slow Due To ArrayIndexOutOfBoundsException

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• /var/log/cb/solr/debug.log has very large number of ArrayIndexOutOfBoundsException error

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -65306 out of bounds for length 66192

• The total aggregate backlog for the EDR server/cluster grows unexpectedly large
• EDR server/cluster may stop ingesting data entirely.

Environment

• EDR Server: 7.3.0 - 7.5.2

Cause

Solr cannot handle more than a certain amount of tokens. There must be a frequent number of events coming in with too many tokens.
The overwhelming error logging caused "document writing threads" to slow down to queue-up stack trace that needs to be logged with each error. 

Resolution

1. Edit the file log4j2.xml.template under /etc/cb/solr/ on all nodes(Master + Minion)

vi log4j2.xml.template

2. Locate the below line in log4j2.xml.template

<AsyncLogger name="com.carbonblack.cbfs.model.solrfields" level="warn" />

3. Add the line given below next to com.carbonblack.cbfs.model.solrfields logger

<AsyncLogger name="com.carbonblack.cbfs.solr.CbProcessUpdateRequestProcessorBase" level="off" />
<AsyncLogger name="com.carbonblack.cbfs.solr" level="off" />

4. Save the file and exit
5. Restart the EDR cluster
 

Additional Information

• This was corrected in 7.6.0 with a new tokenizer. If you are still experiencing this message after 7.6.0, please create a case with support for investigation.