Internet Slow Connection Problem - ICAP connections stuck in the "Reading" state, in CAS.

Products

• ProxySG Software - SGOS

Issue/Introduction

Customer reports that web pages open very slowly or do not even load.

From checking the EventLog s from the Proxy, the  “Maximum concurrent HTTP client worker limit of 100000 reached” error is seen.

As mitigation, the customer bypasses sending traffic to the CAS, from the Proxy, and problem has been solved.

 

Resolution

From investigating the "ca" logs, you would see lots and lots of the "failed to update,Reason: Failed to download patterns: Unexpected HTTP status downloading from URL 'https://subscription.es.bluecoat.com/kaspersky86/patterns' (403)" error.

This may impact user experience and the users may experience delays or disruptions in accessing websites and web services through the ProxySG appliance.

To resolve this, please ensure to disable SSL interception for https://subscription.es.bluecoat.com on the ProxySG, in policy. Also, configure the firewall to exempt CAS from SSL Intercept. Where the CAS accesses the Internet through the ProxySG appliance, collecting a PCAP on the Proxy, with filter set to the CAS IP address, would help further isolate the possible root cause for the above error. From the capture collected, it's important to see, from Wireshark: Statistics > Conversation > IPv4 that there is communication with subscription.es.bluecoat.com (168.149.132.102).

With the above communication confirmed, you may do a Force Update Now, in the CAS appliance. See the steps below.

1. CAS Management Console > Services tab > AV Patterns.

2. Press either Update Now or Force Update Now.

Now, when ICAP connections are stuck in the "Reading" state in Symantec Content Analysis Server (CAS), it typically indicates a problem with communication between the Symantec CAS and the ICAP clients (such as a proxy server or content scanning application). This issue can disrupt the flow of data and cause delays in processing requests. Here are some common causes and troubleshooting steps to address the problem:

• ICAP Service Unavailable: Check if the ICAP service is running and available on the designated port. If the ICAP service is down or not listening on the expected port, connections will remain stuck in the "Reading" state.

• Firewall or Network Connectivity: Ensure that there are no firewall rules or network issues blocking communication between the Symantec CAS and the ICAP clients (Proxy). Verify that the necessary ports are open and accessible.

• ICAP Client Configuration: Review the configuration of the ICAP client (e.g., proxy server) to ensure it is correctly set up to communicate with the Symantec CAS using the appropriate ICAP settings. Here, ensure the correct (recommended) Maximum Allowable ICAP connections are configured in the ICAP service, on the ProxySG appliance. Please refer to the Tech. Article with the URL below, for the recommended values, per CAS model. https://knowledge.broadcom.com/external/article/168737/recommended-icap-connections-on-proxysg.html 

• Resource Constraints: Check the resource utilization on the Symantec CAS server. High CPU or memory usage can lead to delays in processing ICAP requests and result in connections getting stuck in the "Reading" state.

• Logs and Error Messages: Examine the logs and error messages generated by both the Symantec CAS and the ICAP clients. They may provide valuable insights into the root cause of the issue. Please note that we could not work with the uploaded sysinfo file, as it is reported as incomplete. Please refer to the Tech. Article with the URL: https://knowledge.broadcom.com/external/article/166686/download-diagnostic-logs-manually-from-e.html, to collect the sysinfo snapshot for the date of the reported issue and upload the same to the ticket, only for validation checks.

• Restart Services: Sometimes, restarting the ICAP service and related components can resolve connectivity or processing issues.

Lastly, we recommend to implement the ICAP best Practice. In doing so, we recommend to implement following the best practice policy template detailed in the resource with URL below.

https://techdocs.broadcom.com/content/dam/broadcom/techdocs/symantec-security-software/web-and-network-security/proxysg/common/SG_CA_ICAP_Best_Practice_CPL_v1.5_IS_Advanced.txt

In addition, details are available in KB articles such as TECH242686, which describes slowness/latency when turning on ICAP scanning:

https://knowledge.broadcom.com/external/article?legacyId=TECH242686

Note:

Scanning relatively large objects, scanning objects over a smaller bandwidth pipe, or high loads on servers might cause connection timeouts and disrupt the user experience. To prevent such timeouts, you can allow data trickling (data transfer at a very slow rate) to occur. The appliance begins serving server content without waiting for the ICAP scan result. To maintain security, the full object is not delivered until the results of the content scan are complete and the object is determined to not be infected. To take care of this, implementing data trickling is recommended. For the details, please refer to the resource doc. with the URL below.

https://techdocs.broadcom.com/content/dam/broadcom/techdocs/symantec-security-software/web-and-network-security/proxysg/common/ICAP_BP.pdf 

You may also want to consider implementing Deferred Scanning. The deferred scanning feature helps to avoid network outages due to infinite streaming. Infinite streams are connections such as webcams or Flash media (traffic over an HTTP connection) that conceivably have no end. Characteristics of infinite streams may include no content length, slow data rate, and long response time. Because the object cannot be fully downloaded, the ICAP content scan cannot start; however, the connection between the appliance and Content Analysis remains open, causing a wastage of finite connection resources. With deferred scanning, ICAP requests that are unnecessarily holding up ICAP connections are detected and deferred until the full object has been received. For more details, please refer to the same resource doc. with the URL above.

The SSL interception policy used has a direct impact on the amount of content-scanned traffic. Non-intercepted HTTPS traffic is not subject to any ICAP processing, including content-scanning. For example, enabling SSL interception in a network with a 50% SSL encryption rate will double the amount traffic to be scanned by Content Analysis. Symantec recommends that you review the usage statistics of the attached Content Analysis instances before enabling SSL interception on a proxy deployment with activated content scanning.