Data Loss Prevention Network Prevent for Web ICAP Performance Tuning

Products

• Data Loss Prevention
• Data Loss Prevention Network Monitor and Prevent for Email and Web

Issue/Introduction

An approximate summary for tuning Network Prevent for Web (ICAP).  This guide is intended as a ballpark estimate, and you should reach out to professional services for environment specific tuning. 

Resolution

Tuning options

Proxy servers:

Proxy servers ICAP policy should be aligned with the capabilities of the web prevent server.  Consider the following:

• The proxy server's ICAP connection pool should match the available connections on the Web Prevent server pool (Maximum number of requests + Connection Backlog). 
  
  For example, if you have Load balanced 4 web prevent servers, each with 24 maximum number of requests and 24 backlog, and 2 proxy servers to send ICAP traffic to them, each proxy server should be sending 96 connections total (((24 + 24) * 4)/2)
  
  
• The proxy server should only send packets with a content length equal or greater than Request Filtering "ignore packets smaller than" field in the enforce configure page for each Web Prevent server. 
  
  
• The proxy should only send methods which are a concern for data loss.  Typically, those are limited to POST, and PUT.  See the http RFC 7231 for more detailed information on what each kind of method is and make a determination for which of these actually represent a potential avenue for data loss in your environment
  
  
• Work to Identify if there are any internal only sites which are not an avenue for data loss.  These sites do not need to have data monitored by Data Loss Prevention.

DLP Web Prevent Servers:

DLP Network Prevent for Web (Web Prevent) ICAP performance is dependent on the number of CPU cores in the Web Prevent system, the policies enabled in the environment and on the performance of the proxy that feeds data to Web Prevent via ICAP.

Here is a summary of the tuning options that are available, and how they interact with the performance of the Web Prevent system:

• Number of cores available = Number of CPUs (sockets) x Number of Cores per CPU (socket)
  
  
• MessageChain.NumChains = 1 x Number of cores available.
  In some scenarios, 2x Number of cores may be acceptable. Get a baseline with 1 x available cores before testing with 2 x available cores. For Virtual Machines, use 0.80 x Number of cores, rounded-up.
  
  
• MessageChain.CacheSize = Same value as MessageChain.NumChains.
  
  
• Maximum Number of Requests = Default is 25.
  
  
• Connection Backlog: = Same value as Maximum Number of Requests.
  
  
• Request Filtering = Default value is 4096. Minimum recommended value is 2048.
  
  

Explanation of tuning options:

• For tuning purposes, a MessageChain is a single complete detection engine pipeline, which examines the content sent by the ICAP proxy server. 
  
  
• The size of the ICAP "ignore" filter for requests can be configured through enforce under the configure page of the web prevent server.  The smaller the size, the higher the CPU utilization. The default size is 4096 bytes.
  
  
• To properly tune the environment, it will be helpful to know the detection time for every single ICAP request.  Refer to the DLP Help Center (broadcom.com) on how to enable Detection Trace Logging.
  
  
• It is very important to match the proxy server's maximum number of concurrent ICAP connections to the Web Prevent's setting ("Maximum Number of Requests" in the user interface). If the proxy tries to send more connections than the Web Prevent server is configured to accept, this will result in either a service interruption (i.e. "503 Server overloaded" error)* or a "fail open" scenario in which some traffic passes without being examined by DLP.  Equally we don't want to exceed the number of connections as this will result in the DLP server not being used to its full capacity. 
  
  
• The primary constraint on ICAP Prevent performance is the memory limitations of the Java Virtual Machine (JVM). Although previous 32-bit JVM maximum addressable memory restrictions no longer apply in 64-bit systems, it's still possible to exceed the Java Heap. For more information on DLP memory usage and how to resolve issues, see How to address out of memory errors ( OOM ). There is a further requirement for a contiguous block of memory for Java heap space, which is allocated at startup. In practice, based on the default allocation to the BoxMonitor.FileReaderMemory in DLP, the effective size of a single contiguous block of memory is approximately 1.3GB.
  
  
• For best practices and methodology of tuning prevent servers, please see the guides at this page: Symantec™ Data Loss Prevention Network Monitor and Prevent Performance Sizing Guidelines (broadcom.com).
  
  
• *With the default settings, Web Prevent will handle a maximum of 25 active requests and may buffer up to 30MB of content in memory for each request. This accounts for about 750MB of Java heap space (memory usage). The default settings are adequate to allow additional headroom for other functions and may be adjusted based on the observed load in the environment and performance of each server. For more information on this specific topic, see Web Prevent Returning "503 Service Overloaded" to Upstream ICAP Client (broadcom.com).

Additional Information

Note that in addition to the above parameters consideration should also be given to the tuning of detection server java heap memory and also the BoxMonitor.FilereaderMemory settings. In general with the Filereader.MaxFileSize setting at it's default of 30M an allows of 1GB per MessageChain should be more than sufficient for FilereaderMemory, for example with 16 message chains an allocation of 16GB for FilereaderMemory should be allowed. If the MaxFileSize setting is increased to 100M or more then the memory requirements for each MessageChain that is configured will increase exponentially. Current versions of DLP have a "slider" bar for maximum message size which in the configuration screen of the detection server which is constrained by the available hardware, if you find you cannot increase the slider further then you likely need to increase the memory available to the server. Note when calculating memory capacity always allow at least GB for the OS platform.

Also if it is observed that the DetectionServer process is consuming almost all of its allocated java heap memory this is a cause for concern, when java runs low on memory it attempt to perform an activity known as garbage collection to free up more memory, this is a very CPU intensive process and will often  cause a significant drop in detection performance. There you should ensure java always has ample available memory.