I have created a white list of URLs. It is working for explicit request while transparent does not work - why?

Products

• Advanced Secure Gateway Software - ASG
• ProxySG Software - SGOS

Issue/Introduction

Scenario:

• Explicit and Transparent mix deployment
• Created white list are working for users when they point their browser via the proxy.  It is failing when they on transparent deployment (e.g. bridging/inline, WCCP)
• Request are SSL.  Verified that port 443 is set to intercept, transparent under Proxy Services

Resolution

Explanation:

You are seeing a normal behavior if the request is https over transparent with no SSL interception happening and a Server Name Indication extension is not included in the Client Hello packet.

For Explicit https request, browsers or the application does HTTP "CONNECT" method where the domain name is visible on the request which can be seen by the proxy.

Here is an example of explicit HTTPS request:

No.     Time        Source                Destination           Protocol Info

    396 11.565418s  192.168.0.191         192.168.0.5           TCP      61598 > http-alt [SYN] Seq=0 Win=8192 Len=0 MSS=1260 WS=8 SACK_PERM=1

    397 11.565873s  192.168.0.5           192.168.0.191         TCP      http-alt > 61598 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 SACK_PERM=1 WS=6

    398 11.565933s  192.168.0.191         192.168.0.5           TCP      61598 > http-alt [ACK] Seq=1 Ack=1 Win=66560 Len=0

    399 11.566089s  192.168.0.191         192.168.0.5           HTTP     CONNECT bto.bluecoat.com:443 HTTP/1.1    <<<<<<<<<<   Look

    400 11.566312s  192.168.0.5           192.168.0.191         TCP      [TCP Window Update] http-alt > 61598 [ACK] Seq=1 Ack=1 Win=4097472 Len=0

    401 11.623771s  192.168.0.5           192.168.0.191         TCP      http-alt > 61598 [ACK] Seq=1 Ack=206 Win=4097472 Len=0

    402 11.692975s  192.168.0.5           192.168.0.191         HTTP     HTTP/1.1 200 Connection established

    403 11.693573s  192.168.0.191         192.168.0.5           TLSv1    Client Hello

    405 11.773777s  192.168.0.5           192.168.0.191         TCP      http-alt > 61598 [ACK] Seq=40 Ack=389 Win=4097472 Len=0

    406 11.782341s  192.168.0.5           192.168.0.191         TLSv1    Server Hello

Frame 399 shows the HTTP "CONNECT" method after the TCP handshake.

When the browsers or the applications are requesting content towards an HTTPS site transparently via the proxy, URLs are encrypted on HTTPS payload. Without SSL interception or the presence of the SNI extension, URLs can not be seen and would not match https URL created on your white list.
 
Here is an example of Transparent HTTP request:

No.     Time        Source                Destination           Protocol Info

   1279 45.863481s  192.168.0.191         199.91.134.151        TCP      62057 > https [SYN] Seq=0 Win=8192 Len=0 MSS=1260 WS=8 SACK_PERM=1

   1280 45.949138s  199.91.134.151        192.168.0.191         TCP      https > 62057 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 SACK_PERM=1 WS=0

   1281 45.949219s  192.168.0.191         199.91.134.151        TCP      62057 > https [ACK] Seq=1 Ack=1 Win=66560 Len=0

   1282 45.949421s  192.168.0.191         199.91.134.151        TLSv1    Client Hello  <<<<<<<<<Look

   1283 46.036899s  199.91.134.151        192.168.0.191         TLSv1    Server Hello

   1284 46.036977s  199.91.134.151       192.168.0.191        TCP     [TCP segment of a reassembled PDU]

Frame 1282 shows SSL/TLSv1 "CLient Hello" after the TCP Handshake.  It is a normal behavior for browser not to send clear text on the wire for HTTPS site when it (browser) is not proxy aware.

NOTE:  Without SSL interception, Explicit can not match HTTPS URLs if the URLs includes URL path.

Good to read SSL interception articles:

165599 - Configure the SSL proxy on the ProxySG for transparent interception and authentication using an SSL certificate issued from a Microsoft PKI server

166116 - How do I filter non-SSL Intercepted HTTPS traffic on a Transparently deployed Proxy