EDR: Minion Fails to Ingest Data Because Datastore Thread Stuck in Loop
search cancel

EDR: Minion Fails to Ingest Data Because Datastore Thread Stuck in Loop

book

Article ID: 289013

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

  • Minion(s) ingest ZERO events and create SOLR cores with no data
  • PSQL on the Master server has the following error in /var/log/cb/pgsql/postgresql_<date>.txt
    2019-06-20 00:00:10 UTC [cbfs-http(111967) @ <MINION_IP_ADDRESS>(58542)] ERROR: invalid byte sequence for encoding "UTF8": 0x00
    2019-06-20 00:00:10 UTC [cbfs-http(111967) @ <MINION_IP_ADDRESS>(58542)] STATEMENT: INSERT INTO vt_write_events (timestamp, parent_md5hash, file_md5hash, parent_name, file_name, sensor_group_id) VALUES ( $1, $2, $3, $4, $5, $6 ):

Environment

  • EDR Server: 6.x

Cause

Any SQLException while writing to vt_write_event table in PSQL will cause the thread to be in loop and the datastore service would remain busy and not digest data.

Resolution

  1. Upgrade to a supported version of EDR as this issue has been resolved
  2. As a workaround, restart the cb-datastore service on the Master node, which will drop the "bad event data" and moves on to consume next event. 
service cb-datastore stop
service cb-datastore start