Adding Disk Space for Solr CB Event Data

Products

Carbon Black EDR

Issue/Introduction

/var/cb/data hits 100% capacity.
Retention is too low and files cannot be removed to increase disk space.

Environment

Carbon Black EDR: All Supported Versions
Linux OS: All Supported Versions

Cause

There are multiple reasons for requiring additional disk space for process events:

Increased the number of EDR sensors submitting data.
Increased amount of data being collected.
Security policy increases the days of retention required.
Limited disk space available.

Resolution

Add additional disk space for Solr cbevent cores. Solr uses the new cbevents directories if their name is prefixed with cbevents*.
In this scenario, the current (full) cbevents directory resides in the default folder /var/cb/data/cbevents.
1. Mount the new disk device:
```
mount /dev/sdb1 /cbdata
```
2. Symlink the new cbevents2 folder to the current cbevents directory. The symlink name must start with cbevents and trail with a number:
```
ln -s /cbdata /var/cb/data/solr/cbevents2
```
3. Confirm that the Carbon Black EDR user has write permissions in the mounted directory (/cbdata2)
```
chmod cb:cb /var/cb/data/solr/cbevents2
chown 755 /var/cb/data/solr/cbevents2
```
4. Confirm the directory exists, you should see both cbevents and cbevents2:
```
ls -latr /var/cb/data/solr
```

Additional Information

"We recommend using SSD drives on RAID 5 for all Carbon Black EDR deployments with 16,000 IOPS (250 MiB/s throughput) or equivalent performance characteristics."**
"NAS or NFS are not supported for EDR data."**
"For partitions that require 2 TB of storage space or more, at least five solid-state SAS drives in a RAID5 configuration are required."** RAID 5 allows the addition of multiple drives to increase space while providing a level of failover. Preferred over symlinks.
"The count and size-on-disk of process documents are the key factors that drive performance and storage requirements of a Carbon Black EDR deployment."**
Best practices:
- Consolidate cbevents to one volume when possible.
- Only cbevent data should be stored on symlink partitions.
- If possible, EDR data (/var/cb/data) should reside on a separate volume.
- Symlinks provide a good stop-gap feature to quickly add another disk. RAID 5 volumes provide seamless storage increases.
Symlink reminders:

1. If the target of a symlink is moved or deleted, the symlink becomes "broken," and attempts to access it will result in an error.
2. Ensure that users have appropriate permissions to both the symlink and the target to avoid access issues.
3. Backup and restore operations might not handle symlinks as expected. Some backup tools might follow symlinks, potentially duplicating data.
4. Avoid symlinks within a symlinked directory, they can lead to recursive links which can cause confusion and potential issues. Recursive links might also cause infinite loops when traversing directories.

** OER: https://techdocs.broadcom.com/us/en/carbon-black/edr/carbon-black-edr/7-9-1/cb-edr-server-oer-/GUID-C3E2592A-BDA0-430C-8620-5B9ECFDB43B8-en.html

** How to use a Symlinked location for Event Storage

** How to create a symbolic link to move the /var/log/cb data location