Large enterprise customers operating a secure web gateway deployment for thousands of users may experience problems caused by single clients triggering a very high number of failed authentication attempts which potentially can impact the functionality, performance and stability of their ProxySG.
In some cases, this is caused by misbehaving client software which don´t deal well with ProxySG authentication (HTTP 407 ProxySG authentication) or misconfigured clients which cannot authenticate for other reasons (e.g. hard-coded credentials). This article explains how to use the attack-detection feature to identify such clients, get proactive alerting and if desired get proactive blocking for these clients, preventing further performance impact on the ProxySG.
The appliance prevents attacks by limiting the number of simultaneous TCP connections and/or excessive repeated requests from each client IP address that can be established within a specified time frame. If these limits are met, the appliance either does not respond to connection attempts from a client already at this limit or resets the connection. It can also be configured to limit the number of active connections to prevent server overloading.
If the appliance starts seeing a large number of failed requests, and that number exceeds the configured error limit, subsequent requests are blocked and the ProxySG returns a warning page.
Failed requests, by default, include various HTTP response failures such as 4xx client errors (excluding 401 and 407) and 5xx server errors. The HTTP responses that you want treated as failures can be so defined by creating policy.
If the requests continue despite the warnings, and the rate exceeds the warning limits that have been specified for the client, the client is then blocked at the TCP level.
To identify a failed authentication attempt we can leverage the exception.id.
Here is a high-level description from the CPL guide:
Tests whether the exception being returned to the client is the specified exception. It can also be used to determine whether the exception being returned is a built-in or user-defined exception.
Built-in exceptions are handled automatically by the appliance but special handling can be defined within an <exception> layer. Special handling is most often required for user-defined exceptions.
In case of a failed authentication attempt the exception.id field is set to “authentication_failed”, so we can use this combination to configure a policy and set the desired failure weight (here 1).
Another thing to note is that this condition can only be used within an <exception> layer.
A sample policy could look like this:
By default, the attack detection engine triggers on both request and connection limits as well as on failed requests (e.g. server responds with HTTP 403). We can further tune the policy and disable counting other failure reasons. The following would set failure weight to 0 (=disabled) for policy denies and for HTTP 403 Responses.
Next, we would have to set some useful defaults for attack-detection and enable it.
Go to the CLI via serial-console or SSH, go to enable mode and to “conf t” mode.
!- BEGIN networking
enable-limits ;enable Client Mode Attack Detection
interval 1 ;set measurement interval to 1 minute
no default connection-limit ;disable connection monitoring
default failure-limit 100 ;raise failure limit from 50 to 100
default warning-limit 10 ;keep warning limit at 10
;Optional Parameters to change behaviour
default block-action send-tcp-rst ;send tcp-rest instead of dropping
default unblock-time 5 ;automatic unblock after 5 minutes
default monitor-only ;for piloting, enable monitor-only
Indicates the maximum number of failed requests a client is allowed before the ProxySG starts issuing warnings. Default is 50. This limit can be modified on a per-client basis.
Indicates the number of warnings sent to the client before the client is blocked at the network level and the administrator is notified. The default is 10; the maximum is 100. This limit can be modified on a per-client basis.
Other parameters are explained in the SGOS Administration Guide.
With that configuration a client, which triggers more than 100 authentication failures within a one minute interval will cross the failure limit and receive a warning exception page.
If more than 100 authentication failures are triggered and 10 warnings have been sent to the client (110 failures so far then) the next authentication failure will trigger the warning limit and the client is automatically blocked. No more connection will be accepted for the time period set in “unblock-time”. The default is unlimited. In this example, we have set the unblock-time to 5 minutes. This will be logged in the event log like shown in the following (filtered) event log and the client will be added to the blocked-clients list.
Show Event Log filtered:
VSWG-ESX-U100#(config client)show event-log substring "has exceeded"
2017-03-17 14:27:12+01:00CET "Client 192.168.1.156 has exceeded failure limit " 0 580000:1 Mailed logging.cpp:52
2017-03-17 14:27:38+01:00CET "Client 192.168.1.156, has exceeded warning limit and is now blocked " 0 580000:1 Mailed logging.cpp:52
2017-03-17 14:28:54+01:00CET "Client 192.168.1.41 has exceeded failure limit " 0 580000:1 Mailed logging.cpp:52
2017-03-17 14:28:58+01:00CET "Client 192.168.1.41, has exceeded warning limit and is now blocked " 0 580000:1 Mailed logging.cpp:52
VSWG-ESX-U100#(config client)view blocked
IPv4 clients have been blocked Unblock time
192.168.1.41 2017-03-17 13:33:58+00:00UTC
192.168.1.156 2017-03-17 13:32:38+00:00UTC
Example of Access-Log Entry triggering authentication_failed:
(This is shown for every attempt below the failure limit)
2017-03-17 14:17:37 20 192.168.1.41 - - - - None - authentication_failed DENIED "News/Media" - 407 TCP_DENIED GET - http www.ard.de 80 / - - "curl/7.43.0" 192.168.1.212 1487 121 - "none" "none" 2 6cb477b5130675a2-0000000000014f1f-0000000058cfe481 - -
Example of Access-Log Entry exceeding the failure limit (warning is sent):
(This is shown for every attempt above the failure limit, but before the client is blocked)
2017-03-17 14:17:39 1 192.168.1.41 - - 192.168.1.212 192.168.1.212 Ambiguous%20-%20Special%20Use - client_failure_limit_exceeded PROXIED "News/Media" - 503 TCP_DENIED GET - http www.ard.de 80 / - - "curl/7.43.0" 192.168.1.212 1338 121 - "none" "none" 2 6cb477b5130675a2-0000000000014f24-0000000058cfe483 - -
A client which is on the block list will not trigger any further access logs as the connection attempt will either be dropped or rejected (TCP RST).
Monitoring & Troubleshooting commands and Advanced URLs
|CLI||Advanced URLs on Mgmt Console|
|Clients Blocked||show attack-detection client blocked||/ADP/blocked-clients|
|Client Stats||show attack-detection client statistics
show attack-detection client connections
|Top Concurrent Requests||n/a||/ADP/top-concurrent-requests?count=N|
|Event Log||See Below|
|Unblock Client||Unblock <IP>|
|Block Client||Block <IP>|
For piloting this feature one can enable the monitor-only option, which would allow to safely evaluate the thresholds without locking out users and having to manually release them.
VSWG-ESX-U100#(config client)default monitor-only
Syslog Example for Monitor-Only Mode:
(Note the words “Monitor-only” and ‘would’ here)
2017-03-17 14:37:24+01:00CET "Monitor-only client 192.168.1.41 has exceeded failure limit " 0 580000:1 Mailed logging.cpp:52
2017-03-17 14:37:25+01:00CET "Monitor-only Client 192.168.1.41 has exceeded warning limit and would have been blocked " 0 580000:1 Mailed logging.cpp:52
Notes on field use:
The thresholds used in this article may need to be changed in real-world environments to avoid overblocking. Also note that you may want to exclude high-traffic clients such as terminal servers from these default thresholds (see related KBs at the end of this article). It is recommended to pilot this feature in monitor-only mode and fine-tune failure and warning thresholds before going productive in non-monitor mode.
Other related knowledge Base article:
How Attack-detection works on the ProxySG appliance
Attack Detection failure-limits are exceeded very frequently
How do I disable attack detection connection-limits for a single IP address