ProxySG in Explicit Deployment Sends HTTP 401 Response Code for Authentication

Products

• Advanced Secure Gateway Software - ASG
• ProxySG Software - SGOS
• ISG Proxy

Issue/Introduction

You have an explicit proxy that is sending a 401 to the client machine for authentication.  This response code of 401 is meant to be used by servers (and transparent proxies which mimic server behavior).  An explicit proxy should always send an HTTP code of 407 and perform authentication on the CONNECT request.  In your policy trace you see something similar to:

 MATCH:         authenticate(domain1) authenticate.force(no) authenticate.mode(proxy)
...
connection: service.name=Explicit HTTP client.address=10.7X.XXX.XXX proxy.port=8080 client.interface=0:0.1 routing-domain=default
  location-id=0 access_type=unknown
time: 2019-05-24 11:17:14 UTC
CONNECT tcp://www.example.com:443/
  DNS lookup was unrestricted
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36
user: unauthenticated
authentication status='need_credentials' authorization status='not_attempted'
EXCEPTION(authentication_failed): Authentication failed either because credentials were not provided or they could not be validated
  url.category: none@Policy;none@YouTube;none@Local;none@IWF;Social Networking@Blue Coat
    total categorization time: 0
    static categorization time: 0
server.response.code: 0
client.response.code: 407

 

 MATCH:         authenticate(domain1) authenticate.force(no) authenticate.mode(proxy)
...
connection: service.name=Explicit HTTP client.address=10.7X.XXX.XX proxy.port=8080 client.interface=0:0.1 routing-domain=default
  location-id=0 access_type=unknown
time: 2019-05-24 11:17:14 UTC
unknown ssl://www.example.com:443/
  DNS lookup was unrestricted
origin server next-hop IP address=XX.XX.XX.XX
user: name="DOMAIN1\user_name" realm=DOMAIN1_DIRECT
authentication start 0 elapsed 0 ms
authorization start 0 elapsed 0 ms
authentication status='none' authorization status='none'
client.host: (rdns resolution: lookup restricted by policy)
  url.category: none@Policy;none@YouTube;none@Local;none@IWF;Social Networking@Blue Coat
    total categorization time: 0
    static categorization time: 0
application.name: Facebook

MATCH:         authenticate(domain1) authenticate.force(no) authenticate.mode(proxy)
...
MATCH:         url.scheme=https url.domain=//example.com/ authenticate(domain1) authenticate.force(no) authenticate.mode(proxy-ip)
...
connection: service.name=Explicit HTTP client.address=10.7X.XXX.XX proxy.port=8080 client.interface=0:0.1 routing-domain=default
  location-id=0 access_type=unknown
time: 2019-05-24 11:17:14 UTC
GET https://example.com/security/hsts-pixel.gif
  DNS lookup was unrestricted
Referer: https://www.example.com/
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36
user: unauthenticated
authentication status='need_credentials' authorization status='not_attempted'
client.host: (rdns resolution: lookup restricted by policy)
EXCEPTION(authentication_failed): Authentication failed either because credentials were not provided or they could not be validated
  url.category: none@Policy;none@YouTube;none@Local;none@IWF;Technology/Internet@Blue Coat;Social Networking@Blue Coat
    total categorization time: 0
    static categorization time: 0
  request.header.Referer.url.category: none@Policy;none@YouTube;none@Local;none@IWF;Social Networking@Blue Coat
    total categorization time: 0
    static categorization time: 0
server.response.code: 0
client.response.code: 401

 

Cause

On explicit proxies a connection goes through 3 stages (both in the packet capture and when it is processed in policy):

CONNECT tcp:// - This stage is when the client tells the proxy which domain to initiate the SSL handshake with.
unknown ssl:// - This stage is when the client sent the Client Hello and the proxy received it and is actively negotiating the handshake with the server.
GET https:// - This stage is when the SSL handshake completed and Application Data (HTTP within SSL) is flowing through bi-directionally between Client <---> Proxy and Proxy <---> Server.

You need to match the same authentication rule for all 3 of those requests to not break authentication context; primarily the match needs to be the same during CONNECT tcp:// and GET https://.  unknown ssl:// cannot be authenticated (due to the nature of not being able to inject an HTTP response code into an SSL session before it is established / intercepted) so it just takes whoever the authenticated user was from CONNECT tcp://.

If you have different matches between these 3 rules or force the proxy to deauthenticate the client only for one of those stages, you will lose authentication context.  Once the unknown ssl:// stage completes a 407 cannot be injected, so if there is a match on an authentication rule the only available option for the proxy is to send a 401.

Resolution

You need to take a policy trace and see which rules are being matched for the authentication.  In our case we had a different match on a rule containing the url.scheme= condition in the policy trace which was causing the proxy to reauthenticate the client via a 401.

Remove / edit the rules to ensure no processing stage overwrites the authentication rule in such ways.