ICAP 503 errors seen by users and web prevent has a constant buildup of connections

Products

• Data Loss Prevention

Issue/Introduction

Web Prevent 503 service overloaded errors impacting environment.  While DLP doesn't show any related errors in the logs, we do see a buildup of connections on the web prevent server.

Netstat shows a ton of LAST_ACK messages

Cause

A 503 error occurs when going over these limits configured for the WebPrevent/ICAP connections:

Request (16) + Backlog (16) = 32.  The 33rd connection will results in a 503 error. 

 

Resolution

This is an error with the kind of load balancing taking place.  In an F5, its called "One-Connect" profile.  The purpose of this profile is to keep connections open and reuse them.  While this is positive in many networking scenarios, with DLP web prevent this causes a buildup of stale connections which may or may not have traffic flowing.  As new connections come in, the load balancer will continue to create new connections which DLP will try to handle.  Eventually, it will get to a point where there are so many potential connections which might have traffic that DLP can't handle them.  These connections will return a 501 and/or a 503 error.

 

Disable this profile so that Connections are created and closed as traffic comes and goes.  we do not want to re-use the connections. 

Additional Information

https://datatracker.ietf.org/doc/html/rfc793

 

Page 22, TCP/IP Transition State Diagram