CB Response: Upgrade stalls on Migrating Events

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

During upgrade of the Response Server, the upgrade sits at "Migrating Events" indefinitely.

Environment

• CB Response Server: 6.X

Cause

• This issue was caused by corrupted Solr Cores.
• /var/log/cb/solr/startup.out shows:

  corruptIndexException: checksum failed (hardware problem?)

• Solr debug logs shows errors: 

  java.lang.ArrayIndexOutOfBoundsException

Resolution

1. Stop Response Server services: https://community.carbonblack.com/t5/Knowledge-Base/Cb-Response-How-to-restart-server-services/ta-p/41294
2. Move the Solr cores from /var/cb/data/solr5/cbevents/ to a different directory outside of /solr5.
3. Run: 

   /usr/share/cb/cbupgrade

4. Solr will recreate a new core.
5. Migration of event data should complete within minutes.

Additional Information

The integrity of the cores marked as "corrupted" may be unrecoverable depending on where the corruption takes place.