EDR: Server will not start - Solr read timed out
search cancel

EDR: Server will not start - Solr read timed out

book

Article ID: 290527

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

  • Service startup hangs on Solr
  • Solr log shows
<warning>  cb.utils.solr_client - Failed request http://127.0.0.1:8080/solr/admin/cores?action=STATUS&wt=json&indexInfo=true: HTTPConnectionPool(host='127.0.0.1', port=8080): Read timed out. (read timeout=60)
  • job-runner.log shows
carbonblack.inl.gov cb-sensorservices[21972]: cb.core.config.active_grid_config - Key 0 not found in hazelcast
  • Event retention settings in /etc/cb/cb.conf increased beyond default settings

Environment

  • EDR Server: All versions (formerly CB Response)

Cause

There are too many cores for Solr to load before timing out

Resolution

The number of active cores must be reduced manually 
  1. Stop Solr
service cb-solr stop
  1. Create a backup directory outside of /var/cb/data/solr*
  2. Move older cores from /var/cb/data/solr*/cbevents* to the backup directory
  3. If running in RHEL or CentOS 7.x, verify all services are stopped
  4. Start services
service cb-enterprise start

 

Additional Information

  • The recommended number of Solr cores at any time is up to 12 cores or 30 days worth of data, whichever is smaller
  • If more than 30 days of cores are required, consider cold storage or forwarding events to a SIEM