CB Response: Startup fails to proceed past "Waiting for cb-solr to build the terms dictionary"

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Startup fails to proceed past indexing and eventually returns to command prompt

  Starting cb-solr:                                          [  OK  ]
  
     Waiting for cb-solr to build the terms dictionary.
  
     Depending on index size this may take a while.............................
  
  <returns to command prompt without error>

• /var/log/cb/solr/debug.log (on standalone server or minion(s) which failed) shows the following error

  [ERROR] - from org.apache.solr.core.SolrCore in main
  
  null:org.apache.solr.common.SolrException: Found multiple cores with the name [cbevents_<timestamp1>], with instancedirs [/var/cb/data/solr5/cbevents/cbevents_<timestamp2>] and [/var/cb/data/solr5/cbevents/cbevents_<timestamp3>]
  
      at org.apache.solr.core.CoreContainer.checkForDuplicateCoreNames(CoreContainer.java:537)

Environment

• CB Response Server: 6.1.0 - 6.1.1
• Linux: All Supported Versions

Cause

This is due to a rare issue with Solr core rollover identified as CB-13790.

Resolution

$#% WARNING: Do not delete cores unless you are certain this solution is appropriate.  If you are not certain, please log a support ticket for assistance $#% 

1. Find the currently active core (the writer core)

   • find /var/cb/data/solr5/cbevents -type f -name core.properties -exec grep -H 'name=writer' {} \;

2. If the writer core is the most recent core, do not continue and log a support ticket
3. Check all later cores to confirm that they are invalid/empty

   • du -h --max-depth=1 /var/cb/data/solr5/cbevents

4. Delete any directories which start with "cbevents_", have a timestamp after the writer core identified in Step 1, and show only 24k of space used

   • rm -rf /var/cb/data/solr5/cbevents_<timestamp>

5.  Restart Cb Response Services - https://community.carbonblack.com/t5/Knowledge-Base/Cb-Response-How-to-restart-server-services/ta-p/41294

Additional Information

• This issue was fixed in CB Response Server 6.1.2. Upgrading will prevent this issue from occurring again. 
• Be certain to only delete cores that occur after the writer core and reflect only 24k of space used. Deleting any other cores will result in data loss.