URL Categorization Issues with WebPulse and BCWF on ProxySG/EdgeSWG and Advanced Secure Gateway
Article ID: 169134
Updated On:
Products
Issue/Introduction
Sometimes you can see some URLs are not categorized in access log as below.
[14/Jun/2016:14:00:36 +0900] 118 client_IP 200 TCP_NC_MISS 18995 301 GET http host1 80 / - - - - host1 text/html;%20charset=UTF-8 - "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" OBSERVED "none" - Proxy IP x.x.x.x
[14/Jun/2016:14:01:54 +0900] 24 client_IP 304 TCP_NC_MISS 165 1046 GET http host2 80 /include/js/script.js - - - - host2 - http://host2/login/login/ "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" OBSERVED "none" - Proxy IP x.x.x.x
You will get same result if you test these URLs using Content Filtering Diagnostics (Management console > Configuration > Content Filtering > General > Diagnostics).
The result will be something like “Pending / None / None”.
However when you test these URLs on Symantec Site Review (https://sitereview.symantec.com), they are categorized.
Cause
These sites are not in the local categorization database. The reported categories from sitereview.bluecoat.com are fetched in real-time.
Web sites categorized by WebPulse are not added to the categorization database automatically. There are various processes to categorize web sites in the background based on the WebPulse data (in addition to human analysts), and it may take some time for any specific site to be added to the Blue Coat Web Filter (BCWF) database for your appliance to retrieve..
If you are using https://sitereview.symantec.com to check the categorization, you can see two kinds of note on the URLs:
- WebPulse categorization: The Site displays "rated by WebPulse system" when a URL is categorized in real-time as below.
The page you want reviewed is X (Check another site)
This page is currently categorized as <the categories> This page was rated by our WebPulse system
- Database categorization: When a URL is in the categorization database, the site displays last reviewed date instead as below.
The page you want reviewed is X (Check another site)
This page is currently categorized as <the categories> Last Time Rated/Reviewed: > n days
Resolution
Enable ‘Perform dynamic categorization’ (Management console > Configuration > Threat Protection > WebPulse > Dynamic Categorization > Perform dynamic categorization) in ProxySG or Advanced Secure Gateway appliance.
If you don’t want to enable ‘Perform dynamic categorization’ due to some reason like internal security policy, you can request a site to be included in BCWF database manually via WebPulse Site Review (https://sitereview.symantec.com).
Feedback
