Server Upgrade Hangs on Migrating Solr Events

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Query Solr database shows that cores are loaded
• Minimal I/O or CPU activity occurs
• The system startup hangs on migrating events for hours
• /usr/share/cb/cbupgrade --check shows

  • Carbon Black Enterprise Response Server data schema must be upgraded from <ORIGINAL VERSION> to <UPGRADE VERSION>

    
     

Environment

• EDR Server: 6.x and higher (formerly CB Response)

Cause

/var/cb/data/SCHEMA_VERSION file did not update properly

Resolution

1. Edit /var/cb/data/SCHEMA_VERSION 
2. Change the product line to match the <UPGRADE VERSION> listed
   • Ex. product = 7.1.1.200416.1627
3. Restart services

Additional Information

• In some situations, the server license may also reset. The current license will need to be re-applied if this happens 
• For other reasons that the migrating events can hang the /var/log/cb/solr/debug.log can be checked for errors during upgrade
• Solr reader cores can be queried to verify that events have migrated by running the following on affected servers

  • curl http://localhost:8080/solr/reader/select?q=*:*&wt=xml&indent=true&facet=false&&rows=0