EDR: How to Clear Disk Space Safely
search cancel

EDR: How to Clear Disk Space Safely

book

Article ID: 285712

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Disk has filled or is close to filling for /var. (95% or higher) 

Environment

  • EDR Server: All Versions

Resolution

These files are safe to delete. 

  1. Delete any HPROF files that are taking up space
    find /var/log/cb -type f -iname *.hprof* -delete -exec echo "Deleted" {} \;
  2. Delete old logs that may not have been removed
    find /var/log/cb/ -mindepth 1 -type f -mtime +7 -delete -exec echo "Deleted" {} \;
  3. Truncate any deleted files that still have an open file descriptor taking up space
    for i in `find /proc/*/fd -ls 2> /dev/null | awk '/deleted/ {print $11}';`; do truncate -s 0 $i; done

Please read and confirm before running

  1. If you are using the event forwarder and outputting to a file, these are not automatically cleaned up. Delete any older than the days you want to keep, where mtime +7 will remove the files older than 7 days. If you wish to keep the files, move to a separate storage location instead. See: EDR: Disk Full Due to JSON Files for a suggested cron job. 
    find /var/cb/data -type f -mtime +7 -iname *event_bridge* -delete \; -exec echo "Deleted" {} \;
  2. Modulestore: Remove physical binaries (.zip) that are stored locally older than your retention days, or manually set how long you want to keep. See ** in additional notes before running
    • Option 1 - Use retention day setting
      d=$(grep 'MaxEventStoreDays' /etc/cb/cb.conf | awk -F '=' '{print$2}';); `find /var/cb/data/modulestore/ -type f -mtime +$d -iname *.zip* -delete \; -exec echo "Deleted" {} \;`;
    • Option 2 - Manually set days, where mtime +60 will delete older than 60 days
      find /var/cb/data/modulestore/ -type f -mtime +60 -iname *.zip* -delete \; -exec echo "Deleted" {} \;

Event/Binary Data - WARNING

  1. Determine what data directory is taking up the most space under /var/cb/data/. The first output is your total directory size
    declare -a arr=("/var/cb/data" "/var/cb/data/solr*/" "/var/cb/data/modulestore/" "/var/cb/data/solr*/cbmodules/" "/var/cb/data/solr*/cbevents" "/var/cb/data/solr*/cbevents/cbevents_*"); for d in ${arr[@]}; do du -h $d --max-depth=0;done
    • /var/cb/data/modulestore. This is your physical binaries. See the last section for clean up
    • /var/cb/data/solr*/cbmodules. This is your binary metadata (signature, publisher, product name, etc.) - Binary search page
    • /var/cb/data/solr*/cbevents/cbevents_*. This is your process events - Process Search page
  2. If cbmodules is large, please review an enable the cronjob if not done already. By default, these are not removed. EDR: How to Enable Automated Cbmodule Purging
    • Do NOT delete anything within this path. This is a single Solr core holding all binary metadata
  3. For cbevents, first determine if you have cold cores stored
    find /var/cb/data/solr/cbevents -mindepth 1 -maxdepth 1 -type d '!' -exec test -e "{}/core.properties" ';' -print
    
    • If these cores need to be kept, they should be moved to a new storage location outside of the /var/cb partition
    • If these cores can be deleted, run the following, replacing with the core directory name:
      rm -rf /var/cb/data/solr/cbevents/cbevents_<put_core_date_here>
  4. If space is still needed, confirm the oldest active core and remove it. 
    find /var/cb/data/solr/cbevents/cbevents_* -type f -printf '%T+ %p\n' | sort | head -n 1 | awk -F "/" '{print$1,$7}'
    • This will delete the event data that was captured during the time of that core being hot (writer). Solr will need to be stopped first as the data is loaded into memory. Replacing with the core directory name. 
      /usr/share/cb/cbservice cb-solr stop
      rm -rf /var/cb/data/solr/cbevents/cbevents_<put_core_date_here>
      /usr/share/cb/cbservice cb-solr start

Restart Services

  • If you have experienced a disk full situation, Solr and other services may not recover gracefully and ingestion can stop. Restarting services insures that the product is able to continue working as it should
    Standalone:
    /usr/share/cb/cbservice cb-enterprise restart
    
    Cluster via Primary:
    /usr/share/cb/cbcluster stop && /usr/share/cb/cbcluster start

Additional Information

 

  • By default the product will delete the oldest core once disk space hits 90% on the partition holding /var/cb/. It can go over slightly as the job runs every 15 min.
  • During optimization (starts at midnight), it is normal for disk space to get close to 100% full. This is due to the copying of the index files into smaller segments to optimize the performance and space of the core. Once optimization is complete, the unoptimized core is removed and disk space will return to normal. Optimization can take a few hours depending on the size of the core and memory allocated to Solr. 


Article Reference:

** var/cb/data/modulestore holds your physical binaries. These are pulled when downloading a binary zip file from the console. If you are sharing binaries with Alliance, these can be safely removed. 

  • These can be kept around for a few reasons
    • You are not sharing with Alliance. The files will stay locally for download purposes, however this means you are holding onto old binaries that may not be useful any longer and take up disk space
    • KeepAllModuleFiles is set to 1. This setting will not allow the code to delete the local binaries. A clean up job will be needed outside the product
    • A binary file was set to uploaded in Postgres, however they were not successfully removed. 
    • If you wish to keep these, it's suggested to create a separate partition that is mounted to /var/cb/data/cbmodules to not affect Event and Binary metadata available disk space. 
  • Removing these does not affect the binary search page
  • Removing these will generate a 404 if you try to download the file itself from the binary page.