Queued ICAP, Slowness or latency

Products

• ProxySG Software - SGOS

Issue/Introduction

With the ICAP response modification enabled (anti-virus enabled), the Edge SWG (ProxySG) seems to slow down. That slowness seems to increase over time up to a point where the Edge SWG (ProxySG) no longer serves objects.

Seeing many ICAP queued sessions on the Edge SWG (ProxySG).

Seeing many things being scanning in the CAS device under Statistics Tab > Concurrent Connection with the duration high in the milliseconds seeming like the files are stuck in scanning.

Cause

Anti-virus scanning can potentially increase request / response latency as said traffic is now being scanned for malicious content while also being evaluated by the current proxy policy. This is expected behavior. However, there are circumstances that can introduce a more serious issue if latency is found to have increased over time.

Possible Cause(s)

Saturation of the available ICAP connection channels between the Edge SWG (ProxySG) and the ICAP Client/Server appliance (Content Analysis, ProxyAV, DLP, etc...)

• For example, if there are 50 connections available between the Edge SWG (ProxySG) and ICAP Client/Server, it's possible that some of these connections are in use by streaming media, stock tickers or download-related traffic that never terminate. This can cause those ICAP connections to be held up by the Content Analysis system when these files are seen or considered streaming by the Content Analysis.
• You can verify the ICAP connection status, including queued ICAP connections, on the Edge SWG (ProxySG) via Management Console > Statistics Tab > ICAP(6.5.x) or Content Analysis(6.6.x or later)
• You can also verify currently queued object counts via the proxy sysinfo (https://proxyIPaddr:8082/sysinfo) by searching for the keywords "queued_transaction_count" associated with your specific ICAP service.
• For queued connections when sending ICAP externally to Content Analysis, the ICAP service MAX ICAP connection configured should be evaluated. Rule of thumb being: to divide the MAX Connections the Content Analysis can support by the number of proxies sending to it. Always placing the ICAP service in Service Group and using the Service group in the policy for rotation of services. Here is the Article on the MAX ICAP connections recommended and MAX supported per Content Analysis device (Excludes Advanced Secure Gateway Internal ICAP processing): Recommended ICAP conncetions in ProxySG and Maximum ICAPs supported on Content Analysis models

Network outage or connection conflict between the Edge SWG (ProxySG) and the ICAP Client/Server

Conflict at the ICAP scanning appliance end

ICAP scanning enabled

Resolution

Possible Workaround(s) / Solution(s)

Install the ICAP Best Practices via either a proxy VPM CPL layer or the proxy Local file (preferred method)

• The Best Practices is simply serves the purpose of a "guide" based on criteria Symantec has observed amongst a variety of topologies. The Best Practices script typically needs to be modified or "tweaked" per customer environment.
• The Best practices alleviates known ICAP related conflicts related to file size, known incompatibilities / culprits (Streaming media, updates files, etc..) based on destination or protocol.
• Keep note, Prior to 6.5.9.x, that if Malware Scanning is enabled (Configuration > Threat Protection > Malware Scanning),  the ICAP Best Practices are ignored. The ICAP Best Practices + Malware bypass file below, includes a script at the beginning of the file which addresses this known behavior, which can also be removed per customer preference. If running 6.5.9.x or later, the ICAP best Practice.txt should be the only one needed.

NOTE: As of 6.5.9.x and later, the ICAP Best Practice + Malware Bypass CPL code to stop malware scanning to evaluate other policy is no longer required. Please verify on a case by case basis to make sure. Running the ICAP Best Practice.txt File below should be tested first.(Preferred CPL without the Malware Bypass CPL).

Bypass ICAP scanning for known destinations where previous scanning issues have been detected using VPM Cache layer rules. 

In most instances scanning of the following categories should be bypassed:

Audio/Video Clips
Content Servers
Internet Telephony
Mixed Content/Potentially Adult
News/Media
Non-Viewable/Infrastructure
Radio/Audio Streams
TV/Video Streams

NOTE: We now Officially are moving away from Malware Scanning solution, we now recommend enabling the response scanning function in a Web Cache Layer and use the New ICAP Best Improvements:  Secure Web Gateway - Content Analysis Policy Best Practices Improvement