ICAP queued connections observed on Edge SWG (ProxySG) due to Misbehaving/Infinite stream URLs

Products

• Data Loss Prevention Enforce
• ProxySG Software - SGOS
• Data Loss Prevention Network Monitor and Prevent for Web
• ISG Proxy
• ISG Content Analysis

Issue/Introduction

If you observe "ICAP Queued Connections" on Edge SWG (ProxySG) where queued connections were observed in Data Loss Prevention (DLP) (Request Mode) or Content Analysis (Response Mode), where increase in queued connections to large number impacts Edge SWG (ProxySG) to becomes unresponsive with not accepting further new connection request from clients.

Microsoft Azure support DLP.

Environment

Proxy performing ICAP request scanning using DLP or Proxy performing ICAP response scanning using CAS/AV.

DLP Network Prevent for Web, aka "Web Prevent" / Content Analysis 

Cause

It has been observed that if incoming request is something "infinite stream" (i.e. client uploading streaming data to OCS), those requests gets stuck between Proxy and DLP, where those requests could cause "Queued" connections on Proxy. This is expected behavior as Proxy and DLP/CAS device can not determine end of the data stream.

For Example, we have recently observed that, Request to/from "Microsoft Azure" (one example given below) causes the request to get stuck in "transferring state" and causes high amount of queued connections on proxy:

http://52.174.255.158/$servicebus/webstream/1688722a-4857-4317-aa32-6659beba5408/

Above URL looks simple, but it has embedded "webstream" in it.

Resolution

To avoid Queued Connections on Proxy, it is recommended to not send requests with "infinite stream" to the DLP/CAS/AV [Where request of this type does not know where to Terminate (infinite stream)] and causes Loop between Proxy & DLP.

All ICAP Connections configured on ProxySG gets used up and no more further connections are available to serve new requests.

 

How to Troubleshoot ICAP Queued Connections on ProxySG Caused

If you are sending "HTTP/Main" logs from proxy to Reporter, then you can generate a report on Reporter base on Custom Date, Time and Method [PUT, POST].

Look for reports based on "Request" Field and Check for the "Highest Requests", based on that you can determine what "website" or "ServerIP" were requested the most (during queued connections ). Based on type of request if it is something "webstream" , you can make decision and should be able to determine the request causing problems.

 

How to prevent "ICAP Queued Connections" on ProxySG [Request Mode]

If you notice end users are complaining with slowness or degradation in performance while accessing websites:

1. Login to proxy management console, confirm if you are seeing "ICAP Queued Connections", under Statistics ---> Content Analysis --> Duration : Last Hour / Last Day (As per issue timing). See if there are any Queued Requests [Queued Requests will be notified with color pink]
2. if the Queued connection counts are still increasing , then you can determine which requests might be causing this queued, by going to

Statistics ---> Sessions --> Active Sessions ---> Proxies Sessions

Filter : ICAP, REQMOD and Click on Show

Search based on "Duration", if you notice any requests which are stuck in "Transferring" or "Scanning" when you hover mouse to "I" (internet content adoptation protocol). Those are the requests , which are causing queued connections and for some reason are not able to free up. [Due to type of request something like infinite streaming or infinite connection]

To Avoid Queued Connections caused due to Below Request which is related to Microsoft Azure, as we know its issue with request, kindly add below CPL code on ProxySG to avoid queueing:

;=========================================================================================================
<cache>
url.host.is_numeric=yes url.path.substring="servicebus/webstream" request.icap_service(no) response.icap_service(no)
;=========================================================================================================

url.host.is_numeric=yes , is because the IP Address is full CIDR , which keeps on changing, so we are not sending this request for scanning based on specific path in URL.

 

Attached is the latest ICAP best practice that include implementing REQMOD, DLP best practice.