EDR: How to manually optimize a core

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Improve query performance by manually optimizing a core in Solr with large segment counts. For example, segment counts above 50 might lead to significant query performance issues.

Environment

• EDR Server: 6.x and Higher

Resolution

1. Determine the number of segments to use for the manual optimization command
   1. Find the size of the core

du -h /var/cb/data/solr5/cbevents/<core_name>

2. Divide the size in GB by 10 to get a suggested maxSegments value to try. Cores smaller than 200GB should be optimized to 10 segments.
3. Manually optimize each core substituting <core_name> for the core name and <N> for the number of segments

curl 'http://localhost:8080/solr/<core_name>/update?optimize=true&maxSegments=<N>&waitFlush=false'

4. If the optimization fails due to OutOfMemory exceptions, you can increase the maxSegments value and try step 2 again. If the optimization was successful, you can also try reducing the maxSegments and repeating step 2 again to try to further optimize the core on a second pass.

Additional Information

• Consult with Carbon Black Technical Support before manually optimizing cores
• Cores will optimize quicker if the other EDR services have been stopped and only cb-solr is running
• The default core segment count after optimization is 10
• Cores larger than 200GB might not be able to be optimized down to 10 segments due to memory constraints
• The "writer" core is generally not optimized and expected to have larger than 10 segments before rollover
• Larger segment counts will affect search performance. Ideally the segments will be as small as possible; cores should never be optimized below 10 segments
• Cores should automatically be optimized. Manual optimization should only be done if performance issues are present after optimization failed