Truncate any deleted files that still have an open file descriptor taking up space
for i in `find /proc/*/fd -ls 2> /dev/null | awk '/deleted/ {print $11}';`; do truncate -s 0 $i; done
Please read and confirm before running
If you are using the event forwarder and outputting to a file, these are not automatically cleaned up. Delete any older than the days you want to keep, where mtime +7 will remove the files older than 7 days. If you wish to keep the files, move to a separate storage location instead. See: EDR: Disk Full Due to JSON Files for a suggested cron job.
Modulestore: Remove physical binaries (.zip) that are stored locally older than your retention days, or manually set how long you want to keep. See ** in additional notes before running
Determine what data directory is taking up the most space under /var/cb/data/. The first output is your total directory size
declare -a arr=("/var/cb/data" "/var/cb/data/solr*/" "/var/cb/data/modulestore/" "/var/cb/data/solr*/cbmodules/" "/var/cb/data/solr*/cbevents" "/var/cb/data/solr*/cbevents/cbevents_*"); for d in ${arr[@]}; do du -h $d --max-depth=0;done
/var/cb/data/modulestore. This is your physical binaries. See the last section for clean up
/var/cb/data/solr*/cbmodules. This is your binary metadata (signature, publisher, product name, etc.) - Binary search page
/var/cb/data/solr*/cbevents/cbevents_*. This is your process events - Process Search page
If space is still needed, confirm the oldest active core and remove it.
find /var/cb/data/solr/cbevents/cbevents_* -type f -printf '%T+ %p\n' | sort | head -n 1 | awk -F "/" '{print$1,$7}'
This will delete the event data that was captured during the time of that core being hot (writer). Solr will need to be stopped first as the data is loaded into memory. Replacing with the core directory name.
If you have experienced a disk full situation, Solr and other services may not recover gracefully and ingestion can stop. Restarting services insures that the product is able to continue working as it should
By default the product will delete the oldest core once disk space hits 90% on the partition holding /var/cb/. It can go over slightly as the job runs every 15 min.
During optimization (starts at midnight), it is normal for disk space to get close to 100% full. This is due to the copying of the index files into smaller segments to optimize the performance and space of the core. Once optimization is complete, the unoptimized core is removed and disk space will return to normal. Optimization can take a few hours depending on the size of the core and memory allocated to Solr.
Article Reference:
** var/cb/data/modulestore holds your physical binaries. These are pulled when downloading a binary zip file from the console. If you are sharing binaries with Alliance, these can be safely removed.
These can be kept around for a few reasons
You are not sharing with Alliance. The files will stay locally for download purposes, however this means you are holding onto old binaries that may not be useful any longer and take up disk space
KeepAllModuleFiles is set to 1. This setting will not allow the code to delete the local binaries. A clean up job will be needed outside the product
A binary file was set to uploaded in Postgres, however they were not successfully removed.
If you wish to keep these, it's suggested to create a separate partition that is mounted to /var/cb/data/cbmodules to not affect Event and Binary metadata available disk space.
Removing these does not affect the binary search page
Removing these will generate a 404 if you try to download the file itself from the binary page.