Web Prevent Returning "503 Service Overloaded" to Upstream ICAP Client

Products

• Data Loss Prevention Network Prevent for Web

Issue/Introduction

Web Prevent can accept as many connections as the sum of Max Request, Max Response and Backlog connections as defined under Server -> Configure -> ICAP (tab) -> Connection:

 

An additional 1 connection allowance is made for the local, internal health check. See server setting: Icap.DisableHealthCheck, default value of false.

Options Response

 

If the ICAP client (e.g. proxy) sends an "options request", we send an options response that includes a "MAX_CONNECTIONS_HEADER" header with the max REQMod and max RESPMod connections that we support. The values in this response are calculated by dividing the given maximum number by Icap.LoadBalanceFactor. If Icap.LoadBalanceFactor is greater than the maximum number defined, 1 is returned.

 

If supported by the client, this allows it to dynamically tune their connections to our connections. If this is not supported, then the upstream clients need to have their connections tuned with the above arithmetic in mind.

Cause

An HTTP Response code of 503 Service Overloaded is returned to the ICAP client when any connection attempt would cause all current ICAP connections to exceed:

• Reqmod connections (Maximum Number of Requests)
• + Respmod connections (Maximum Number of Responses)
• + Connection Backlog
• + 1 (the plus 1 is hardcoded to allow for the local, internal health checker task).

Resolution

Ensure that the ICAP clients connecting to the Web Prevent never exceed the combined maximum number of Request, Response and Backlog connections as defined on the Web Prevent under Server -> Configure -> ICAP (tab) -> Connection.

Best practice is to either have the ICAP clients dynamically tune their outbound connections by sending an options request, or hard code their Maximum outbound Request and Response connections to be the same as our corresponding Maximum settings. If there is an external, third party health check on the ICAP port, that also needs to be accounted for.