Cbupgrade Loops with 500 Error on cbfeeds Core with Domain Search Errors.

Products

• Carbon Black EDR (formerly Cb Response)
• Carbon Black Hosted EDR (formerly Cb Response Cloud)

Issue/Introduction

Search containing dot (.) in Threat Intelligence page generates a red message will pop up with a 500 server error in 7.7.0 or higher. 

During cbupgrade the cbfeeds core migration loops with a 500 error on 7.9.1 or higher. 

• Error displayed during cbupgrade attempt. 

   request to http://localhost:8080/solr/cbfeeds/select failed with error <500 - Server Error>. Parameters:- wt => 'json' - rows => 0

• Error seen in /var/log/cb/solr/debug.log during the upgrade attempt. 

      "metadata":["error-class","org.apache.solr.core.SolrCoreInitializationException","root-error-class","java.lang.IllegalArgumentException"],
      "msg":"SolrCore 'cbfeeds' is not available due to init failure: Error opening new searcher",
      "trace":"org.apache.solr.core.SolrCoreInitializationException: SolrCore 'cbfeeds' is not available due to init failure: Error opening new searcher

Environment

• Carbon Black EDR: 7.7.0 and Higher. 

Cause

The schema for the cbfeeds Solr core was updated and the already existing feed data is still on the old schema

Resolution

Follow below procedure in case you have the possibility to rollback the upgrade by restoring previous snapshots, otherwise, please contact Broadcom support.

In case the Servers are still not upgraded, then proceed with performing below procedure:

1. Stop the services: How to Start, Stop and Restart EDR Application Services
2. Backup the old feed core

   mv /var/cb/data/solr/cbfeeds /var/cb/cbfeeds_backup

3. Generate a new feed core

   /usr/share/cb/virtualenv/bin/python -m cb.maintenance.cbstartup.main --stage startup

4. Start only these services

   /usr/share/cb/cbservice cb-redis start
   /usr/share/cb/cbservice cb-solr start

5. Run the indexer to ingest the data from the back up core to the new schema

   /usr/share/cb/cb-solr-reindexer --solr-port 8080 --solr-address 127.0.0.1 cbfeeds /var/cb/cbfeeds_backup/data/index

6. Stop the running services

   /usr/share/cb/cbservice cb-solr stop
   /usr/share/cb/cbservice cb-redis stop

7. Start the EDR application services
8. Verify the feeds are displaying in the console and the search is now working. Remove the backup file

   rm -rf /var/cb/cbfeeds_backup

 

Additional Information

• This can take several minutes to complete the reindexing. Approximately 1 minute per 1 million documents. Use the following command can be used to get a count of the documents. 

  curl -s "http://localhost:8080/solr/cbfeeds/select?q=*:*&rows=0&wt=json" | /usr/share/cb/virtualenv/bin/python3 -m json.tool | grep numFound