Detect Authentication Request Floods with the Attack Detection on proxy and Advanced Secure Gateway
Article ID: 169384
Updated On:
Products
Issue/Introduction
Large enterprise customers operating a secure web gateway deployment for thousands of users may experience problems caused by single clients triggering a very high number of failed authentication attempts which potentially can impact the functionality, performance and stability of their Edge SWG (ProxySG).
In some cases, this is caused by misbehaving client software which don´t deal well with Edge SWG (ProxySG) authentication (HTTP 407 authentication) or misconfigured clients which cannot authenticate for other reasons (e.g. hard-coded credentials). This article explains how to use the attack-detection feature to identify such clients, get proactive alerting and if desired get proactive blocking for these clients, preventing further performance impact on the Edge SWG (ProxySG).
The appliance prevents attacks by limiting the number of simultaneous TCP connections and/or excessive repeated requests from each client IP address that can be established within a specified time frame. If these limits are met, the appliance either does not respond to connection attempts from a client already at this limit or resets the connection. It can also be configured to limit the number of active connections to prevent server overloading.
If the appliance starts seeing a large number of failed requests, and that number exceeds the configured error limit, subsequent requests are blocked and the Edge SWG (ProxySG) returns a warning page. Failed requests, by default, include various HTTP response failures such as 4xx client errors (excluding 401 and 407) and 5xx server errors. The HTTP responses that you want treated as failures can be so defined by creating policy. If the requests continue despite the warnings, and the rate exceeds the warning limits that have been specified for the client, the client is then blocked at the TCP level.
Resolution
Configuration
To identify a failed authentication attempt we can leverage the exception.id.
Here is a high-level description from the CPL guide:
exception.id
Tests whether the exception being returned to the client is the specified exception. It can also be used to determine whether the exception being returned is a built-in or user-defined exception.
Built-in exceptions are handled automatically by the appliance but special handling can be defined within an <exception> layer. Special handling is most often required for user-defined exceptions.
In case of a failed authentication attempt the exception.id field is set to “authentication_failed”, so we can use this combination to configure a policy and set the desired failure weight (here 1).
Another thing to note is that this condition can only be used within an <exception> layer.
A sample policy could look like this:
<Exception>
exception.id=authentication_failed attack_detection.failure_weight(1)
By default, the attack detection engine triggers on both request and connection limits as well as on failed requests (e.g. server responds with HTTP 403). We can further tune the policy and disable counting other failure reasons. The following would set failure weight to 0 (=disabled) for policy denies and for HTTP 403 Responses.
<exception>
exception.id=policy_denied attack_detection.failure_weight(0)
<proxy>
http.response.code=403 attack_detection.failure_weight(0)
Next, we would have to set some useful defaults for attack-detection and enable it.
Go to the CLI via serial-console or SSH, go to enable mode and to “conf t” mode.
!- BEGIN networking
attack-detection ;mode
client ;mode
enable-limits ;enable Client Mode Attack Detection
interval 1 ;set measurement interval to 1 minute
no default connection-limit ;disable connection monitoring
default failure-limit 100 ;raise failure limit from 50 to 100
default warning-limit 10 ;keep warning limit at 10
;Optional Parameters to change behaviour
default block-action send-tcp-rst ;send tcp-rest instead of dropping
default unblock-time 5 ;automatic unblock after 5 minutes
default monitor-only ;for piloting, enable monitor-only
exit
exit
Failure limit:
Indicates the maximum number of failed requests a client is allowed before the Edge SWG (ProxySG) starts issuing warnings. Default is 50. This limit can be modified on a per-client basis.
Warning limit:
Indicates the number of warnings sent to the client before the client is blocked at the network level and the administrator is notified. The default is 10; the maximum is 100. This limit can be modified on a per-client basis.
Other parameters are explained in the SGOS Administration Guide.
With that configuration a client, which triggers more than 100 authentication failures within a one minute interval will cross the failure limit and receive a warning exception page.
If more than 100 authentication failures are triggered and 10 warnings have been sent to the client (110 failures so far then) the next authentication failure will trigger the warning limit and the client is automatically blocked. No more connection will be accepted for the time period set in “unblock-time”. The default is unlimited. In this example, we have set the unblock-time to 5 minutes. This will be logged in the event log like shown in the following (filtered) event log and the client will be added to the blocked-clients list.
Show Event Log filtered:
VSWG-ESX-U100#(config client)show event-log substring "has exceeded"
2017-03-17 14:27:12+01:00CET "Client x.x.x.x has exceeded failure limit " 0 580000:1 Mailed logging.cpp:52
2017-03-17 14:27:38+01:00CET "Client x.x.x.x has exceeded warning limit and is now blocked " 0 580000:1 Mailed logging.cpp:52
2017-03-17 14:28:54+01:00CET "Client x.x.x.x has exceeded failure limit " 0 580000:1 Mailed logging.cpp:52
2017-03-17 14:28:58+01:00CET "Client x.x.x.x has exceeded warning limit and is now blocked " 0 580000:1 Mailed logging.cpp:52
VSWG-ESX-U100#(config client)
VSWG-ESX-U100#(config client)view blocked
IPv4 clients have been blocked Unblock time
x.x.x.x 2017-03-17 13:33:58+00:00UTC
x.x.x.x 2017-03-17 13:32:38+00:00UTC
VSWG-ESX-U100#(config client)
Access-Logging:
Example of Access-Log Entry triggering authentication_failed:
(This is shown for every attempt below the failure limit)
2017-03-17 14:17:37 20 x.x.x.x - - - - None - authentication_failed DENIED "News/Media" - 407 TCP_DENIED GET - http www.example.com 80 / - - "curl/7.43.0" x.x.x.x 1487 121 - "none" "none" 2 6cb477b5130675a2-0000000000014f1f-0000000058cfe481 - -
Example of Access-Log Entry exceeding the failure limit (warning is sent):
(This is shown for every attempt above the failure limit, but before the client is blocked)
2017-03-17 14:17:39 1 x.x.x.x - - x.x.x.x Ambiguous%20-%20Special%20Use - client_failure_limit_exceeded PROXIED "News/Media" - 503 TCP_DENIED GET - http www.example.com 80 / - - "curl/7.43.0" x.x.x.x - "none" "none" 2 6cb477b5130675a2-0000000000014f24-0000000058cfe483 - -
A client which is on the block list will not trigger any further access logs as the connection attempt will either be dropped or rejected (TCP RST).
Monitoring & Troubleshooting commands and Advanced URLs
| CLI | Advanced URLs on Mgmt Console | |
| Clients Blocked | show attack-detection client blocked | /ADP/blocked-clients |
| Client Stats | show attack-detection client statistics |
/ADP/clients |
| Client Connections | show attack-detection client connections |
/TCP/Users |
| Top Requests | n/a | /ADP/top-requests?count=N |
| Top Failures | n/a | /ADP/top-failures?count=N |
| Top Concurrent Requests | n/a | /ADP/top-concurrent-requests?count=N |
| Debug Log | n/a | /ADP/Debug |
| Event Log | See Below | |
| Unblock Client | Unblock <IP> | |
| Block Client | Block <IP> |
Monitor-Only Feature:
For piloting this feature one can enable the monitor-only option, which would allow to safely evaluate the thresholds without locking out users and having to manually release them.
CLI:
VSWG-ESX-U100#(config client)default monitor-only
ok
Syslog Example for Monitor-Only Mode:
(Note the words “Monitor-only” and ‘would’ here)
2017-03-17 14:37:24+01:00CET "Monitor-only client x.x.x.x has exceeded failure limit " 0 580000:1 Mailed logging.cpp:52
2017-03-17 14:37:25+01:00CET "Monitor-only client x.x.x.x has exceeded warning limit and would have been blocked " 0 580000:1 Mailed logging.cpp:52
Notes on field use:
The thresholds used in this article may need to be changed in real-world environments to avoid over blocking. Also note that you may want to exclude high-traffic clients such as terminal servers from these default thresholds (see related KBs at the end of this article). It is recommended to pilot this feature in monitor-only mode and fine-tune failure and warning thresholds before going productive in non-monitor mode.
Search for other related knowledge Base article:
How Attack-detection works on the Edge SWG (ProxySG) appliance
Attack Detection failure-limits are exceeded very frequently
Feedback
