EDR: Corrupted CBmodules on a Minion Stop the Cluster Startup

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Cluster does not complete a start-up due to one minion's service failure between cb-solr and cb-coreservices.
• The Solr logs may contain one of the following entries:

"IndexNotFoundException: no segments* file found LockValidatingDirectoryWrapper"

"SolrCore 'cbmodules' is not available due to init failure: Error opening new searcher"

 

Environment

• EDR Servers: 7.6 and higher

Cause

The cbmodules indexing has become corrupt.

Resolution

Remove the contents of /var/cb/data/solr/cbmodules/ for that one minion.  The Primary server maintains the official copy of the cbmodules indexing and re-populates the minion's cbmodule indexing if they were removed.  This synchronizes the cbmodules indexing across the cluster and the cluster should start.

Additional Information

• Just remove the contents of the cbmodules directory, not the directory itself.
• Consider making a backup of the directory before deleting it just in case it is needed.  It can be removed once the cluster starts and the Primary server properly updates the minion cbmodules directory. 

tar -cvf cbmodulesmeta.tar /var/cb/data/solr/modules/

• The /var/cb/data/solr/cbmodules directory contains the metadata and indexing for the binary files.  The actual binary files are stored in another directory /var/cb/data/cbmodules/ and are unique on each server;  They should not be removed for this fix.
• The replication.properties file is normal in /var/cb/data/solr/cbmodules/data/ directory.