EDR: Unexpected Log Entry or Corrupt Log Causing Solr Not to Start

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Solr hangs while starting due to a corrupt tlog file.
• A similar error message is found to the following example in /var/log/cb/solr/debug.log

  2016-01-01 01:01:01 - [WARN] - from org.apache.solr.update.UpdateLog in coreLoadExecutor-3-thread-1
  Unexpected log entry or corrupt log. Entry=1523java.lang.ClassCastException: null

  
   

Environment

• EDR Server: All versions

Cause

• Usually caused by Solr not being properly shut down while indexing.

Resolution

1. Create a corrupt tlog directory and move all tlogs to this directory

   mkdir /var/cb/data/solr/cbevents/0/data/tlogcorrupt
   mv /var/cb/data/solr/cbevents/0/data/tlog/tlog.* /var/cb/data/solr/cbevents/0/data/tlogcorrupt/

   • Repeat the process for any remaining shards
2. Stop any remaining services
   • Standalone

     service cb-enterprise stop

   • Cluster

     /usr/share/cb/cbcluster stop

3. Remove all cb user owned processes

   killall -KILL -u cb

4. Start the Services
   • Standalone

     service cb-enterprise start

   • Cluster

     /usr/share/cb/cbcluster start

5. Remove the tlogcorrupt folder once services have started

   rm -rf /var/cb/data/solr/cbevents/0/data/tlogcorrupt

   • Repeat the process for any remaining shards

Additional Information

• The tlog are raw documents created for recovery purposes. Since these are now corrupt, they are no longer useful. If they are left behind, solr will take up more disk space than what is specified in /etc/cb/cb.conf.