Web Prevent 503 service overloaded errors impacting environment. While DLP doesn't show any related errors in the logs, we do see a buildup of connections on the web prevent server.
Netstat shows a ton of LAST_ACK messages
A 503 error occurs when going over these limits configured for the WebPrevent/ICAP connections:
Request (16) + Backlog (16) = 32. The 33rd connection will results in a 503 error.
This is an error with the kind of load balancing taking place. In an F5, its called "One-Connect" profile. The purpose of this profile is to keep connections open and reuse them. While this is positive in many networking scenarios, with DLP web prevent this causes a buildup of stale connections which may or may not have traffic flowing. As new connections come in, the load balancer will continue to create new connections which DLP will try to handle. Eventually, it will get to a point where there are so many potential connections which might have traffic that DLP can't handle them. These connections will return a 501 and/or a 503 error.
Disable this profile so that Connections are created and closed as traffic comes and goes. we do not want to re-use the connections.
https://datatracker.ietf.org/doc/html/rfc793
Page 22, TCP/IP Transition State Diagram