Solr corruption prevents services from starting

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• EDR Server services fails during Solr startup, with no services starting after Solr.
• Manual start of services after Solr will work, but no data will be written to the Solr cores.

Environment

• EDR Server:  7.x +

Cause

• Corruption in cbmodules causes Solr to fail in creating searcher.
• Disk I/O errors on the host has been known to cause this.

Resolution

Following commands will list and attempt to repair the corrupted values in cbmodules: 

• # Check for cbmodules index corruption
  /usr/lib/jvm/jre-11/bin/java -cp /usr/share/cb/solr/server/solr-webapp/webapp/WEB-INF/lib/*:/usr/share/cb/solr/server/lib/ext/*:/usr/share/cb/datastore/cbfs-http/* -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /var/cb/data/solr/cbmodules/data/index -fast
  

 

• # Fix cbmodules index corruption
  /usr/lib/jvm/jre-11/bin/java -cp /usr/share/cb/solr/server/solr-webapp/webapp/WEB-INF/lib/*:/usr/share/cb/solr/server/lib/ext/*:/usr/share/cb/datastore/cbfs-solr/* -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /var/cb/data/solr/cbmodules/index -exorcise

 

Additional Information

• Errors seen in /var/log/cb/solr/startup.log:

  "qtp1358444045-154" #154 prio=5 os_prio=0 tid=0x00007fca30075000 nid=0x2ed3 waiting on condition [0x00007fc6c058e000]
     java.lang.Thread.State: TIMED_WAITING (parking)
          at sun.misc.Unsafe.park(Native Method)
          - parking to wait for  <0x00000006fa61cf70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
          at java.lang.Thread.run(Thread.java:748)
  

 

• Error seen in /var/log/cb/solr/debug.log: 

  Caused by: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=0 vs expected header=1071082519 (resource=MMapIndexInput(path="/var/cb/data/solr5/cbmodules/data/index/_jfa.fdt"))
          at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196)
          at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255)
          at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:153)
          at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121)
          at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173)
          at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:117)
          at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
          at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
          at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
          at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:99)
          at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:435)
          at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:100)
          at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
          at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:40)
          at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1660)

 

• Data loss may occur, and the customer should expect that not everything may be recoverable.