FileReader memory and profile native memory

Products

• Data Loss Prevention Enterprise Suite

Issue/Introduction

In the FileReader0.log(prior to 16.1) on the DLP Detection servers we have this line present: 

"INFO: Profile Loaded, Configured FileReader memory and profile native memory total is : 202048.0 MB.

Information related to EDM memory size requirement and impact while loading. 

Resolution

The java heap memory is needed for communication between detection and the native matcher execution is not based on the number of indexes or their size, just based on the number of message chains configured. The overall system memory on the other hand depends on the number of message chains and the total size of all cells of all indexes.

The EDM_Memory_Requirements_Spreadsheet is a tool that you can use to determine the additional system memory needed on the detection server to run your indexes.It is designed to provide an estimation (rounding a little up) of the SYSTEM RAM needed for loading the indexes.

If there are more than 10 EDM indexes then they simply reuse the same spreadsheet to calculate each set of 10 indexes (sorted highest number of cells to lowest) and then sum up the total of the "Required RAM" from each set of 10 indexes. So if there are 11 indexes then the largest 10 indexes will go into the first spreadsheet and the lowest will go into the last spreadsheet. At most the total will be overestimated by 3GB.

The more accurate calculation would be when FileReader actually loads the indexes and provides the memory used by FileReader to load the indexes which includes both the size from the java heap and the native system memory that was the additional impact for loading the indexes. There are other minor memory uses that are not included in this info log record but that is just a few GB. The can take this memory and use that as the additional memory needed on-top of our sizing guidelines. Medium Installation Hardware Recommendations