EDR Server: Only a few cores loaded in Solr

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• UI search only shows events from the last few days
• Querying loaded solr cores only shows a few cbevents* mounted
• /var/cb/data/solr*/cbevents shows many more cbevents* directories
• Server has run out of disk space or is above 90% disk usage

Environment

• EDR Server: All Versions
• /etc/cb/cb.conf value: AlwaysDeleteColdPartitions=False

Cause

• Once the server disk usage passes a certain threshold, Solr will roll over the oldest loaded core. 
• With AlwaysDeleteColdPartitions=False, older cores are unmounted, but will not be removed automatically. So the server never goes below the disk threshold

Resolution

Disk usage needs to be reduced below the MaxEventStoreSizeInPercent (default 90%)

• Remove old cores to a separate storage location
• Increase drive space on server

Additional Information

• It is not recommended that cold cores be stored on the /var/cb partition
• New cores can be remounted manually, but the cb.user_mounted file will need to be removed: https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-to-remount-Cold-Partitions/ta-p/62446
• To view mounted cores, run: 

  • curl "http://127.0.0.1:8080/solr/admin/cores?action=STATUS&wt=json&indexInfo=false&indent=true" | grep name