Upgrade Stalls While Migrating Solr Database

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• While upgrading an EDR server using the cbupdate script, the process hangs while attempting to migrate Solr database cores.
• A seemingly endless row of "dot" [ ......... ] appears on screen.

 

Environment

• EDR: 7.x and Higher

Cause

One or more Solr cores may be corrupted.  

Resolution

There are two possible solutions.

Solution 1: Repair the Core Files

1. Cancel the upgrade process.
2. Ensure all EDR services are fully stopped. 
3. Identify the problematic core inside /var/cb/data/cbevents/
4. Repair the core file using these commands.
5. Restart the upgrade process.

Solution 2: Remove the Core File

1. Cancel the upgrade process.
2. Ensure all EDR services are fully stopped. 
3. Identify the problematic core inside /var/cb/data/cbevents/
4. Remove the core file directory:    # rm -rf /var/cb/data/cbevents/<core file name>  NOTE: Deleting a core file will result in the permanent loss of event data for that time period. 
5. Restart the upgrade process.

Additional Information

• Use the /var/log/cb/solr/debug.log file to help identify problematic core files. Search on the core name within the log file to look for errors. 
• Look for core files that are older than the server's retention time. Those might be problematic. 
• These instructions apply to both primary and secondary servers.