Customer reports that web pages open very slowly or do not even load.
From checking the EventLog s from the Proxy, the “Maximum concurrent HTTP client worker limit of 100000 reached” error is seen.
As mitigation, the customer bypasses sending traffic to the CAS, from the Proxy, and problem has been solved.
From investigating the "ca" logs, you would see lots and lots of the "failed to update,Reason: Failed to download patterns: Unexpected HTTP status downloading from URL 'https://subscription.es.bluecoat.com/kaspersky86/patterns' (403)" error.
This may impact user experience and the users may experience delays or disruptions in accessing websites and web services through the ProxySG appliance.
To resolve this, please ensure to disable SSL interception for https://subscription.es.bluecoat.com on the ProxySG, in policy. Also, configure the firewall to exempt CAS from SSL Intercept. Where the CAS accesses the Internet through the ProxySG appliance, collecting a PCAP on the Proxy, with filter set to the CAS IP address, would help further isolate the possible root cause for the above error. From the capture collected, it's important to see, from Wireshark: Statistics > Conversation > IPv4 that there is communication with subscription.es.bluecoat.com (22.214.171.124).
With the above communication confirmed, you may do a Force Update Now, in the CAS appliance. See the steps below.
1. CAS Management Console > Services tab > AV Patterns.
2. Press either Update Now or Force Update Now.
Now, when ICAP connections are stuck in the "Reading" state in Symantec Content Analysis Server (CAS), it typically indicates a problem with communication between the Symantec CAS and the ICAP clients (such as a proxy server or content scanning application). This issue can disrupt the flow of data and cause delays in processing requests. Here are some common causes and troubleshooting steps to address the problem:
Lastly, we recommend to implement the ICAP best Practice. In doing so, we recommend to implement following the best practice policy template detailed in the resource with URL below.
In addition, details are available in KB articles such as TECH242686, which describes slowness/latency when turning on ICAP scanning:
Scanning relatively large objects, scanning objects over a smaller bandwidth pipe, or high loads on servers might cause connection timeouts and disrupt the user experience. To prevent such timeouts, you can allow data trickling (data transfer at a very slow rate) to occur. The appliance begins serving server content without waiting for the ICAP scan result. To maintain security, the full object is not delivered until the results of the content scan are complete and the object is determined to not be infected. To take care of this, implementing data trickling is recommended. For the details, please refer to the resource doc. with the URL below.
You may also want to consider implementing Deferred Scanning. The deferred scanning feature helps to avoid network outages due to infinite streaming. Infinite streams are connections such as webcams or Flash media (traffic over an HTTP connection) that conceivably have no end. Characteristics of infinite streams may include no content length, slow data rate, and long response time. Because the object cannot be fully downloaded, the ICAP content scan cannot start; however, the connection between the appliance and Content Analysis remains open, causing a wastage of finite connection resources. With deferred scanning, ICAP requests that are unnecessarily holding up ICAP connections are detected and deferred until the full object has been received. For more details, please refer to the same resource doc. with the URL above.
The SSL interception policy used has a direct impact on the amount of content-scanned traffic. Non-intercepted HTTPS traffic is not subject to any ICAP processing, including content-scanning. For example, enabling SSL interception in a network with a 50% SSL encryption rate will double the amount traffic to be scanned by Content Analysis. Symantec recommends that you review the usage statistics of the attached Content Analysis instances before enabling SSL interception on a proxy deployment with activated content scanning.