How to Correct a Corrupted Solr Index

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

How to correct a corrupted Solr Index

• Solr is unable to startup
• Some items in the webui do not appear to be rendering
• All watchlists showing "error"

Environment

• EDR Server: 7.x and Above

Resolution

Run the following command on the backend server seeing the index corruption error message. Replace <corrupt core> with the core seen in the path= error.

• Version 7.x EL7/8

/usr/lib/jvm/jre-11/bin/java -cp /usr/share/cb/solr/server/solr-webapp/webapp/WEB-INF/lib/*:/usr/share/cb/solr/server/lib/ext/*:/usr/share/cb/datastore/cbfs-solr/* -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /var/cb/data/solr/cbevents/<corrupt core>/data/index -exorcise

 

Note: Your CheckIndex path may be different. Use the MMapIndexInput path as a guide to where the corruption path is. 

Additional Information

• -fix can be used in place of -exorcise to see how many indexes will be removed.
• -fast can be used instead of -exorcise to find out if there are multiple issues before waiting a long time the -exorcise option to complete
• Data is non-recoverable in the corrupt state. Running this will allow the most amount of data to be recovered without deleting the entire core. 
• Corruption can occur for multiple reasons including a server crash or image restoration from a snapshot
• Possible error messages seen in /var/log/cb/solr/debug.log

  Insert document exception 
  org.apache.solr.common.SolrException: Exception writing document id 0000b3d0-0000-25d4-01d4-b33ab90c7390-01687b99d538 to the index; possible analysis error. 
  ...
  Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed 
  ...
  Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/cb/data/solr5/cbevents/cbevents_2019_01_22_2015/data/index/_fi2.fnm")