How to control what is sent up to DRTR via CPL?

Products

ISG Proxy

Issue/Introduction

Once configured on an Edge SWG device Webpulse / DRTR sends data upstream to the service per configuration.

How can we precisely control what is sent upstream, and when, using the CPL properties available for Webpulse?

Environment

Edge SWG with the Webpulse feature licensed and enabled

Resolution

The following properties are available if you want to have granular control over what is sent to DRTR / Webpulse services:

We will focus on the categorize options here.

The property webpulse.categorize.mode() is available and implemented in tcp and ssl layers as seen in this policy trace excerpt:

TCP transaction

start transaction -------------------
transaction ID=nnnnn1 type=http.proxy
    CONNECT tcp://example:443/

        <Cache@req-url> [layer 21] [local:26]
 MATCH:         webpulse.categorize.mode(none) 

        <Cache> [layer 23] [local:32]
 MATCH:         webpulse.categorize.send_url(host) 

        <Proxy> [layer 25] [local:39]
 MATCH:         category=none FORCE_DENY 

connection: service.name=Explicit HTTP client.address=... (effective address=...) proxy.port=8080 source.port=... dest.port=8080
time: yyyy-mm-dd hh:mm:ss UTC
CONNECT tcp://example:443/
  DNS lookup was unrestricted
verdict: DENIED: (inherited) Either 'force_deny' or 'force_exception' was matched in policy(handed off) 
  url.category: none
    dynamic categorization was performed: mode=none
      dynamic categorization time: 0
server.response.code: 0
client.response.code: 403

bytes received from client: 96
bytes sent to server      : 0
bytes received from server: 0
bytes sent to client      : 39

  url_categorization complete time: 34
stop transaction --------------------

SSL transaction

transaction ID=nnnnn2 type=ssl.tunnel
*** transaction created with committed exception (policy_denied)
    unknown ssl://example:443/

        <Cache@req-url> [layer 21] [local:26]
 MATCH:         webpulse.categorize.mode(none) 

        <Cache> [layer 23] [local:32]
 MATCH:         webpulse.categorize.send_url(host) 

        <Proxy> [layer 25] [local:39]
 MATCH:         category=none FORCE_DENY 

connection: service.name=Explicit HTTP client.address=... (effective address=...) proxy.port=8080 source.port=... dest.port=8080 
time: yyyy-mm-dd hh:mm:ss UTC
unknown ssl://example:443/
verdict: DENIED: (inherited) Either 'force_deny' or 'force_exception' was matched in policy(handed off) 
    dynamic categorization was performed: mode=none

bytes received from client: 0
bytes sent to server      : 0
bytes received from server: 0
bytes sent to client      : 0

The properties webpulse.categorize.send_headers() and webpulse.categorize.send_url() is available and implemented in http or https layers:

start transaction -------------------
*** transaction created with committed exception (policy_denied)
transaction ID=nnnnn3 type=https.forward-proxy
    GET https://example/test/yyyy/mm/dd/hhmm23utc

  late: <Cache@req-url> [layer 21] [local:26]

        <Cache> [layer 23] [local:32]
 MATCH:         webpulse.categorize.send_url(host) 

        <Proxy> [layer 25] [local:39]
 MATCH:         category=none FORCE_DENY 

connection: service.name=Explicit HTTP client.address=... (effective address=...) proxy.port=8080 source.port=... dest.port=8080
time: yyyy-mm-dd hh:mm:ss UTC
GET https://example/test/yyyy/mm/dd/hhmm23utc
verdict: DENIED: (inherited) Either 'force_deny' or 'force_exception' was matched in policy
    dynamic categorization was performed: mode=realtime
      dynamic categorization time: 130
bytes received from client: 89
bytes sent to server      : 0
bytes received from server: 0
bytes sent to client      : 9288

stop transaction --------------------

When DRTR is used to send the hostname per the above https transaction we can see the data in the OPP Debug logs (abridged logs sorted by time ascending):

Sending to DRTR: /...redacted.../https/example/443/
Request succeeded: DRTR: 'none'=>'none'RL=>'6' : 'example/test/yyyy/mm/dd/hhmm23utc' time (126) .

The DRTR request shows that we only send the url host, whilst the request succeed message shows the full url.

Here are some OPP log samples with the default DRTR configurations (raw logs, sorted by time descending):

HTTPS transaction sending the method, url host and path data:

8688.124 PDW t=nnnnn9 for 7254 Sending to DRTR: /...redacted.../GET/https/example/443/test/yyyy/dd/mm/hhmm43utc

SSL transaction sending the url host data:

8688.039 PDW t=nnnnn6 for 7253 Sending to DRTR: /...redacted.../unknown/ssl/example/443/

TCP transaction sending the http method and

8687.958 PDW t=nnnnn3 for 7252 Sending to DRTR: /...redacted.../CONNECT/tcp/example/443/

And here is another set of similar DRTR OPP logs from a test that was run earlier than the above transactions:

8679.600 PDW t=nnnnn5 for 7247 Sending to DRTR: /...redacted.../GET/https/example/443/test/yyyy/dd/mm/hhmm35utc
8679.516 PDW t=nnnnn2 for 7246 Sending to DRTR: /...redacted.../unknown/ssl/example/443/
8679.345 PDW t=nnnnn1 for 7245 Sending to DRTR: /...redacted.../CONNECT/tcp/example/443/