EDR: Unexpected Log Entry or Corrupt Log Causing Solr Not to Start
search cancel

EDR: Unexpected Log Entry or Corrupt Log Causing Solr Not to Start

book

Article ID: 289923

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

  • Solr hangs while starting due to a corrupt tlog file.
  • A similar error message is found to the following example in /var/log/cb/solr/debug.log
    2016-01-01 01:01:01 - [WARN] - from org.apache.solr.update.UpdateLog in coreLoadExecutor-3-thread-1
    Unexpected log entry or corrupt log. Entry=1523java.lang.ClassCastException: null

     

Environment

  • EDR Server: All versions

Cause

  • Usually caused by Solr not being properly shut down while indexing.

Resolution

  1. Create a corrupt tlog directory and move all tlogs to this directory
    mkdir /var/cb/data/solr/cbevents/0/data/tlogcorrupt
    mv /var/cb/data/solr/cbevents/0/data/tlog/tlog.* /var/cb/data/solr/cbevents/0/data/tlogcorrupt/
    • Repeat the process for any remaining shards
  2. Stop any remaining services
    • Standalone
      service cb-enterprise stop
    • Cluster
      /usr/share/cb/cbcluster stop
  3. Remove all cb user owned processes
    killall -KILL -u cb
  4. Start the Services
    • Standalone
      service cb-enterprise start
    • Cluster
      /usr/share/cb/cbcluster start
  5. Remove the tlogcorrupt folder once services have started
    rm -rf /var/cb/data/solr/cbevents/0/data/tlogcorrupt
    • Repeat the process for any remaining shards

Additional Information

  • The tlog are raw documents created for recovery purposes. Since these are now corrupt, they are no longer useful. If they are left behind, solr will take up more disk space than what is specified in /etc/cb/cb.conf.