proxy authentication pop-ups after migrating to a new proxy, linked with only a specific set of client/users

Products

• ProxySG Software - SGOS

Issue/Introduction

After migration to a new proxy, the following issue was seen.

Behavior: When the user opens the web browser and receives an authentication popup (whereas the method is NTLM) which should not be there (Tested for Both Chrome/Edge). Once the user manually enters it does work but once the user closes the web browser and opens it again authentication is required...

User configuration :

Citrix machine (SBC): The user uses the same terminal as another user, meaning more than 1 user uses the same machine
Authentication method: NTLM
Policy trace -> no policy trace generated for the user, only for IP.

Analysis :

- only 1 specific user is impacted among the others for this machine-> always this one
- We tried with another user on the same machine, same domain -> no issue
- the same policy is installed for both user

- Via LSA/debug: we do not see any authentication test/lookup except when the user succeeded in manually authenticating.
- on the event log: nothing except a "Resetting Schannel due to error" (Yes generally this is the root cause but it does not match at all the timing of the test)
- via pcap file: will not help as the machine is "shared" for many users

Among 3000 users we only have the same 66 users which does.

Are there any historical data about this kind of authentication pop-up?

Resolution

Having investigated the logs, while we do not yet have information about any of the affected users or their IP addresses, we see a lot of authentication failures and Schannel resets. For the authentication failures, see the attached .csv file, for your reference. For the Schannel resets, see the snippet below.

For the description, causes, and resolution guidance for the Schannel error, refer to the Tech. Article with the URL below.

https://knowledge.broadcom.com/external/article/175348/troubleshooting-steps-for-failed-authent.html

Now, for what we already know, concerning the authentication pop-up, refer to the below.

There can be many reasons why users may receive authentication pop-ups. This submission explains only one specific example. 

When NTLM authentication for proxy authentication is employed, authentication pop-ups display when you change the Windows password by password policy.

Condition:
An authentication pop-up box is displayed when the customer changes the Windows password by password policy.
A special application is used to change the Windows password, which causes the authentication pop-up.

Troubleshooting steps:

• To check the authentication password: Change the authentication method for only basic credentials (this option is just for troubleshooting so that you can confirm with a packet capture the actual password being sent from the browser).

• Take a packet capture (PCAP)

Potential Results:
From PCAP: The Windows client used an old password for Proxy authentication. If this is the case, the problem is not caused by the ProxySG but rather the client sending an incorrect password.

Specifically, with Internet Explorer, refer to the Tech. Article with the URL below, for another possible reason for multiple authentication pop-ups. 

https://knowledge.broadcom.com/external/article/173732/multiple-authentication-popups-in-intern.html

For Unexpected authentication pop-ups occurring when using NTLM proxy authentication, refer to the details in the Tech. Article with the URL below. 

https://knowledge.broadcom.com/external/article/167057/unexpected-authentication-popups-occur-w.html

Returning back to the logs, we see lots of authentication made against two domains called "BIL" & "Pine", and looking at the configuration from the sysinfo, we do not seem to see an IWA realm linked with these two domains. Rather we see that the configured authentication realm is linked with a domain called "LDC", where the Schannel reset is happening. See the additionally attached .csv files, for the authentication failures linked with the "BIL" & "Pine" unknown domains.

So, from the investigation done, we see that the authentication failures are linked with the Schannel resets and the "BIL" & "Pine" unknown domains. For these unknown domains, in addition to the other resolution recommendations already communicated, we recommend that you engage your Microsoft AD team and have them investigate the unknown domains, if they are truly unknown, they want to ensure that these domains have no linkage with the ProxySG appliance and it's configured realm.

Also, you should ensure that all affected users are a part of the configured IWA realm and are active. They should not be linked with domains that are not a part of the configured IWA realm. If they are, the Proxy will receive authentication credentials from those users, as delivered from those unknown domains, and as the Proxy delivers the Type 3 NTLM message to the authentication server, it will see those users as unknown, since their domain is not a part of the IWA realm, and will drop the authentication.

Note: From the investigation done, nothing points the issue to the proxy. The causes of the reported authentication pop-up point to the Schannel reset and other environmental factors, already shared. If you would require any further investigation, we will require logs and PCAP, specific to the affected clients/users alongside fresh entire event log and sysinfo, collected after reproducing the issue with a number of the affected clients/users.