EDR: Server will not start - Solr read timed out

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Service startup hangs on Solr
• Solr log shows

<warning>  cb.utils.solr_client - Failed request http://127.0.0.1:8080/solr/admin/cores?action=STATUS&wt=json&indexInfo=true: HTTPConnectionPool(host='127.0.0.1', port=8080): Read timed out. (read timeout=60)

• job-runner.log shows

carbonblack.inl.gov cb-sensorservices[21972]: cb.core.config.active_grid_config - Key 0 not found in hazelcast

• Event retention settings in /etc/cb/cb.conf increased beyond default settings

Environment

• EDR Server: All versions (formerly CB Response)

Cause

There are too many cores for Solr to load before timing out

Resolution

The number of active cores must be reduced manually 

1. Stop Solr

service cb-solr stop

2. Create a backup directory outside of /var/cb/data/solr*
3. Move older cores from /var/cb/data/solr*/cbevents* to the backup directory
4. If running in RHEL or CentOS 7.x, verify all services are stopped
5. Start services

service cb-enterprise start

 

Additional Information

• The recommended number of Solr cores at any time is up to 12 cores or 30 days worth of data, whichever is smaller
• If more than 30 days of cores are required, consider cold storage or forwarding events to a SIEM