Startup Fails to Proceed Past "Waiting for cb-solr to build the terms dictionary"

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Startup fails to proceed past indexing and eventually returns to command prompt.

  Starting cb-solr:                                          [  OK  ]
  
     Waiting for cb-solr to build the terms dictionary.
  
     Depending on index size this may take a while.............................
  

• /var/log/cb/solr/debug.log shows the following error on an eventful node. 

  [ERROR] - from org.apache.solr.core.SolrCore in main
  
  null:org.apache.solr.common.SolrException: Found multiple cores with the name [cbevents_<timestamp_1>], with instancedirs [/var/cb/data/cbevents/cbevents_<timestamp_2>] and [/var/cb/data/cbevents/cbevents_<timestamp_3>]
  
      at org.apache.solr.core.CoreContainer.checkForDuplicateCoreNames(CoreContainer.java:537)

Environment

• Carbon Black EDR Server: All Versions

Cause

• Solr was busy or unresponsive during core rollover.
• Disk full.
• Manual intervention. 

Resolution

 

1. Find the core names that are currently listed as a writer core 

   • find /var/cb/data/solr/cbevents/ -type f -name core.properties -exec grep -H 'name=writer' {} \; | awk -F '/' '{print$7}'

2. Update the core.properties file on only the older cores, the newest core will remain as a writer. Replace 'cbevents_timestamp_' with the older core dates.
   • Template command to use:

     for i in 'cbevents_timestamp_1'; do sed -i "s/writer/$i/" /var/cb/data/solr/cbevents/$i/core.properties

   • Examples of the command.

     Example single core change. 
     for i in 'cbevents_2025_03_12_0114'; do sed "s/writer/$i/" /var/cb/data/solr/cbevents/$i/core.properties; done
     
     Example of multiple core change. 
     for i in 'cbevents_2025_03_12_0114' 'cbevents_2025_03_31_1332'; do sed "s/writer/$i/" /var/cb/data/solr/cbevents/$i/core.properties; done

3. Capture a list of all core sizes. 
   

   • du -sh /var/cb/data/solr/cbevents/* | sort -rh | grep -v 'reader'

4. Delete any cbevents_ directory that shows 24K of spaced used. Keeping the "reader" and the writer (newest core). 

   • rm -rf /var/cb/data/solr/cbevents_<timestamp>

5.  Attempt to start the EDR services again. 

Additional Information

• Be certain to only delete cores that occur after the writer core and reflect only 24k of space used. Deleting any other cores will result in data loss.
• There must always be one writer core.