EDR: How to Correct a Corrupted Solr Index
search cancel

EDR: How to Correct a Corrupted Solr Index

book

Article ID: 285282

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

How to correct a corrupted Solr Index
  • Solr is unable to startup
  • Some items in the webui do not appear to be rendering
  • All watchlists showing "error"

Environment

  • EDR Server: 7.x and Above

Resolution

Run the following command on the backend server seeing the index corruption error message. Replace <corrupt core> with the core seen in the path= error.
  • Version 7.x EL7/8
/usr/lib/jvm/jre-11/bin/java -cp /usr/share/cb/solr/server/solr-webapp/webapp/WEB-INF/lib/*:/usr/share/cb/solr/server/lib/ext/*:/usr/share/cb/datastore/cbfs-solr/* -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /var/cb/data/solr/cbevents/<corrupt core>/data/index -exorcise
 

Note: Your CheckIndex path may be different. Use the MMapIndexInput path as a guide to where the corruption path is. 

Additional Information

  • -fix can be used in place of -exorcise to see how many indexes will be removed.
  • -fast can be used instead of -exorcise to find out if there are multiple issues before waiting a long time the -exorcise option to complete
  • Data is non-recoverable in the corrupt state. Running this will allow the most amount of data to be recovered without deleting the entire core. 
  • Corruption can occur for multiple reasons including a server crash or image restoration from a snapshot
  • Possible error messages seen in /var/log/cb/solr/debug.log
    Insert document exception 
    org.apache.solr.common.SolrException: Exception writing document id 0000b3d0-0000-25d4-01d4-b33ab90c7390-01687b99d538 to the index; possible analysis error. 
    ...
    Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed 
    ...
    Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/cb/data/solr5/cbevents/cbevents_2019_01_22_2015/data/index/_fi2.fnm")