The impact of disabling Dynamic Categorization

Products

• ProxySG Software - SGOS

Issue/Introduction

The customer requested to understand what impact disabling Dynamic Categorization would have on the ProxySG and on existing URL access. The customer also requested to have some statistics for the percentage of URLs sent to WebPulse for rating, with Dynamic Categorization. 

Resolution

Dynamic analysis of content is performed through the WebPulse cloud service and not locally on the ProxySG appliance. There is a small amount of bandwidth used for the round-trip request and response, and a slight amount of time waiting for the service to provide results. As the service is only consulted for URLs that cannot be locally categorized using the Symantec WebFilter database and WebPulse results are cached on the appliance, the user experience is generally not
affected.

To avoid per-request latency, you might want to run dynamic categorization in background mode.

The following diagram illustrates Symantec WebFilter’s content filtering flow when dynamic categorization is employed.

Process Flow

1. (Blue arrow) Client 1 requests a Web page.
2. The ProxySG appliance checks the requested URL against the Symantec WebFilter database for categorization. No match is found.
3. The WebPulse Service returns the categorization of the URL if it has already been determined. If not, WebPulse accesses and analyzes the requested site and returns a real-time categorization if the confidence rating is high enough. If a category cannot be determined automatically with high confidence, the service returns a category unknown status but records the site for future categorization.
4. After the URL is categorized, the policy engine determines if the URL is allowable or not. Steps 5 and 6 describe what happens if the URL is allowable. Step 7 describes what happens if the URL is not allowable.
5. (Blue arrow) The URL is allowed and the request continues to its destination for full 

Dynamic categorization has three states:

• Enabled: The service attempts to categorize unrated websites. This is the default state. When enabled, the ProxySG appliance accesses the WebPulse cloud service for categorizing a requested URL only when it is not available in the Symantec WebFilter database.

• Disabled: If the service is disabled, the ProxySG appliance does not contact the WebPulse service, regardless of any policy that might be installed. The Symantec WebFilter database is consulted for categorization and based on the policies installed on the ProxySG appliance, the requested content is served or denied.

• Suspended: Categorization from the database continues, but the service is no longer employed. This occurs when the installed database is over 30 days old due to the expiration of WebFilter download credentials or network problems. After credentials are renewed or network problems are resolved, the service returns to Enabled.

Now, depending on whether a category is found in BCWF or rating cache, you might see a variety of status results returned, such as None, Pending, Unlicensed, or Unavailable. 

None. No categories are available for the URL. Categories might not be returned because:

• The ProxySG appliance did not get a response from the WebPulse service.
• The WebPulse service was unable to retrieve the requested URL in a timely manner.
• The WebPulse service cannot categorize the request with high confidence.

References to all URLs requested in WebPulse are recorded for future categorization in WebFilter by automated background analysis or human analysis.

Note: Timeout is currently set to three seconds. The average response time for WebPulse to retrieve the content and perform the real-time analysis is under 500 milliseconds.

Pending. The ProxySG appliance continues to service the URL request without waiting for a response from WebPulse.

If a response is received, it is added to the rating cache, so future requests for that same URL will have the appropriate list of categories returned immediately. Reference to the site is recorded for future categorization in the WebFilter database by automated background URL analysis or human analysis.

If a response is not received in a timely manner, or the request results cannot be categorized, nothing is added to the rating cache. 

Note: It is possible that multiple requests for the same content can result in a Pending status if WebPulse has not completed processing the first request before subsequent requests for the same URL are received by the ProxySG appliance.

Unlicensed. A problem exists with the WebFilter license. Here, unrated URLs will not be accessed.

Unavailable. A problem (other than licensing) exists with the local WebFilter database or accessing the WebPulse service. Here, unrated URLs will not be accessed.

Of all the described states, only the "None" and "pending" states could pose a potential security threat.

The "Pending" is assigned to the URL request when dynamic categorization is performed in Background mode when the ProxySG appliance continues to service the URL request without waiting for a response from the WebPulse dynamic categorization service. So, to avoid a potential security hole, do not run dynamic categorization in the background mode.

For the "None" state, a potential security hole can be prevented by configuring the URL categorization rule to "deny" access for any URL request rated as "none".

With policy, access to URLs rated as "Unavailable" can also be denied.

We recommend not running dynamic categorization in the background mode and denying access to URLs rated as "none" & "Unavailable", in policy, to prevent security holes.

Concerning Dynamic Categorization, please be informed that there is no statistical data that may be collected from the ProxySG appliance, as these would be directly WebPulse data. Nonetheless, we are able to collect the drtr debug log (OPP/debug), as shown in the snippet below. This logs and debugs drtr communication between the ProxySG and WebPulse.

For concerns around the WebPulse site review, please route the concerns to the dedicated Broadcom team, following the guidance provided in the article with the URL below.

https://knowledge.broadcom.com/external/article/171535/proxysg-content-filtering-categorizes-a.html

From ProxySG, we are able to collect statistical data for content filtering, as shown in the snippet below.

Ref. docs.:

https://knowledge.broadcom.com/external/article/165330/about-dynamic-categorization-and-how-it.html

SGOS 6.7.x.x Admin Guide.