In last step of IE/Chrome request, browser stops at HTTP/1.1 401 Unauthorized.
Release : 12.8
Component : SITEMINDER SECURE PROXY SERVER
The cause comes down to how Kerberos authentication is performed and how SiteMinder implements the authentication chaining flow. An HTTP server does not send an HTTP response specifically requesting Kerberos authentication. To begin authentication with the web browser the HTTP server sends an HTTP 401 Unauthorized response with a "WWW-Authenticate:" header. The value of the header determines the kinds of authentication mechanisms supported by the web server. In the case of Kerberos the mechanism is "Negotiate", but this includes both Kerberos authentication as well as NTLM authentication.
If the browser can perform Kerberos authentication, then it acquires a Kerberos service ticket to the web server and sends it in an HTTP "Authorization:" header to the web server to be authenticated.
If the browser cannot perform Kerberos authentication (not configured to do so, cannot contact the Kerberos Key Distribution Center (KDC), etc.) then the browser falls back to attempting NTLM. To perform NTLM the browser prompts for username and password. This is the pop up that appears (often mistakenly called a "basic pop up" used for HTTP Basic authentication).
At this point, the browser shows the NTLM authentication pop up and does not automatically move forward, falling back to the HTML form.
If the user enters credentials, authentication will fail since SiteMinder is expecting Kerberos credentials not NTLM. Because Kerberos authentication chaining is configured the browser will then be redirected to the HTML form page.
If the user presses the Cancel button in the pop up dialog, the browser typically sends another request without credentials. Because Kerberos authentication chaining is configured, the first HTTP 401 response includes a cookie (SMKRBCHALLENGE=YES). On the second request, after pressing Cancel without credentials but with the cookie, the browser will be redirected to the HTML form page (and the cookie value cleared).
The SiteMinder web agent code does include an HTML body in the initial HTTP 401 response with an HTML "meta redirect" that immediately reloads the page. The intent is to try a workaround to prevent display of the pop up dialog. If Kerberos is working, then the reload will contain Kerberos credentials. If Kerberos is not working, then the reload will not contain Kerberos credentials but will have the state cookie and the browser will be redirected to the HTML form.
The problem is that the HTML "meta redirect" may or may not be honored depending on browser, browser version, and operating system. Thus the pop up dialog cannot always and consistently be avoided.
Browser support is out of Siteminder product scope. And no one can have full control over external users' browser settings.
It is best to use HTML login form for external users and keep Kerberos for internal users. Or have a main landing page for everyone, and let the users decide what option they would like to do (external vs. internal).
NTLM / Kerberos authentication disabled by default in Incognito mode and guest sessions
Ambient authentication (NTLM/Kerberos) will be disabled by default in Incognito mode and guest sessions in Chrome 81. To revert to the old behavior and allow ambient authentication, use the AmbientAuthenticationInPrivateModesEnabled policy.
Kerberos troubleshooting guide
The sequence of Kerberos Authentication