Are files saved locally after they are extracted?

Products

• Security Analytics

Issue/Introduction

When traffic being captured by Security Analytics triggers a rule hit for any data enrichment provider where a file needs to be extracted, where are those files stored and how long do they stay around?

Resolution

The files that match the enrichment rules and that are extracted are always saved to disk first.  They are saved in the /home/extractor-live directory. 

Depending on the size of the files, they will be automatically cleaned up after 15 or 30 minutes.  Larger files are cleaned up within 30 minutes and smaller files are cleaned up by 15 minutes from the extraction time.

Here is the example of what you see in the logs of this process:

2021-10-20T02:40:02+00:00 hostname extractor_cleanup[88349]: main: expiring manifested files older than 1634696702 : Wed Oct 20 02:25:02 2021
2021-10-20T02:40:02+00:00 hostname extractor_cleanup[88349]: main: deleted 8974 files from 10 manifests
2021-10-20T02:40:02+00:00 hostname extractor_cleanup[88349]: main: expiring non-manifested files and empty directories older than 1634695802 : Wed Oct 20 02:10:02 2021
2021-10-20T02:40:03+00:00 hostname extractor_cleanup[88349]: main: deleted 245 non-manifested files, and 381 empty directories

If a manual extraction is executed, the file is stored in the /home/apache/artifacts directory.