EDR: Event Ingestion Is Slow Due To ArrayIndexOutOfBoundsException
search cancel

EDR: Event Ingestion Is Slow Due To ArrayIndexOutOfBoundsException

book

Article ID: 285781

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

  • /var/log/cb/solr/debug.log has very large number of ArrayIndexOutOfBoundsException error
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -65306 out of bounds for length 66192
  • The total aggregate backlog for the EDR server/cluster grows unexpectedly large
  • EDR server/cluster may stop ingesting data entirely.

Environment

  • EDR Server: 7.3.0 - 7.5.2

Cause

Solr cannot handle more than a certain amount of tokens. There must be a frequent number of events coming in with too many tokens.
The overwhelming error logging caused "document writing threads" to slow down to queue-up stack trace that needs to be logged with each error. 

Resolution

1. Edit the file log4j2.xml.template under /etc/cb/solr/ on all nodes(Master + Minion)
vi log4j2.xml.template

2. Locate the below line in log4j2.xml.template

<AsyncLogger name="com.carbonblack.cbfs.model.solrfields" level="warn" />
3. Add the line given below next to com.carbonblack.cbfs.model.solrfields logger
<AsyncLogger name="com.carbonblack.cbfs.solr.CbProcessUpdateRequestProcessorBase" level="off" />
<AsyncLogger name="com.carbonblack.cbfs.solr" level="off" />
4. Save the file and exit
5. Restart the EDR cluster
 

Additional Information

  • This was corrected in 7.6.0 with a new tokenizer. If you are still experiencing this message after 7.6.0, please create a case with support for investigation.