The customer requested to understand what impact disabling Dynamic Categorization would have on the ProxySG and on existing URL access. The customer also requested to have some statistics for the percentage of URLs sent to WebPulse for rating, with Dynamic Categorization.
Dynamic analysis of content is performed through the WebPulse cloud service and not locally on the ProxySG appliance. There is a small amount of bandwidth used for the round-trip request and response, and a slight amount of time waiting for the service to provide results. As the service is only consulted for URLs that cannot be locally categorized using the Symantec WebFilter database and WebPulse results are cached on the appliance, the user experience is generally not
affected.
To avoid per-request latency, you might want to run dynamic categorization in background mode.
The following diagram illustrates Symantec WebFilter’s content filtering flow when dynamic categorization is employed.
Process Flow
Dynamic categorization has three states:
Now, depending on whether a category is found in BCWF or rating cache, you might see a variety of status results returned, such as None, Pending, Unlicensed, or Unavailable.
None. No categories are available for the URL. Categories might not be returned because:
References to all URLs requested in WebPulse are recorded for future categorization in WebFilter by automated background analysis or human analysis.
Note: Timeout is currently set to three seconds. The average response time for WebPulse to retrieve the content and perform the real-time analysis is under 500 milliseconds.
Pending. The ProxySG appliance continues to service the URL request without waiting for a response from WebPulse.
If a response is received, it is added to the rating cache, so future requests for that same URL will have the appropriate list of categories returned immediately. Reference to the site is recorded for future categorization in the WebFilter database by automated background URL analysis or human analysis.
If a response is not received in a timely manner, or the request results cannot be categorized, nothing is added to the rating cache.
Note: It is possible that multiple requests for the same content can result in a Pending status if WebPulse has not completed processing the first request before subsequent requests for the same URL are received by the ProxySG appliance.
Unlicensed. A problem exists with the WebFilter license. Here, unrated URLs will not be accessed.
Unavailable. A problem (other than licensing) exists with the local WebFilter database or accessing the WebPulse service. Here, unrated URLs will not be accessed.
Of all the described states, only the "None" and "pending" states could pose a potential security threat.
The "Pending" is assigned to the URL request when dynamic categorization is performed in Background mode when the ProxySG appliance continues to service the URL request without waiting for a response from the WebPulse dynamic categorization service. So, to avoid a potential security hole, do not run dynamic categorization in the background mode.
For the "None" state, a potential security hole can be prevented by configuring the URL categorization rule to "deny" access for any URL request rated as "none".
With policy, access to URLs rated as "Unavailable" can also be denied.
We recommend not running dynamic categorization in the background mode and denying access to URLs rated as "none" & "Unavailable", in policy, to prevent security holes.
Concerning Dynamic Categorization, please be informed that there is no statistical data that may be collected from the ProxySG appliance, as these would be directly WebPulse data. Nonetheless, we are able to collect the drtr debug log (OPP/debug), as shown in the snippet below. This logs and debugs drtr communication between the ProxySG and WebPulse.
For concerns around the WebPulse site review, please route the concerns to the dedicated Broadcom team, following the guidance provided in the article with the URL below.
https://knowledge.broadcom.com/external/article/171535/proxysg-content-filtering-categorizes-a.html
From ProxySG, we are able to collect statistical data for content filtering, as shown in the snippet below.
Ref. docs.:
https://knowledge.broadcom.com/external/article/165330/about-dynamic-categorization-and-how-it.html
SGOS 6.7.x.x Admin Guide.