Users are redirected to the wrong authentication scheme.
This most often happens when the product is working as designed, but the configuration and policies are yielding unexpected results. When a user is challenged for credentials, the authentication scheme is determined by the realm that is protecting the requested resource. The realm that the requested resource is mapped to is determined by a combination of the requested URI along with the resolved AgentName. The AgentName for any given request is determined by either mapping the hostname from the request to an AgentName defined in the Agent Configuration Object for the Web Agent handling the request, or if the hostname from the request does not match any defined AgentName, the DefaultAgentName will be used.
The most surefire way to understand why a particular authentication scheme was invoked for a given request is to study the Web Agent logs. The error log is useful to view the ACO parameters, particularly the AgentName and DefaultAgentName parameters. The Web Agent trace log will show the incoming request, including the resolved AgentName and URI (Resolved URL). In a small environment this may be sufficient information to find the Realm via the Admin UI and confirm the authentication scheme. In larger environments this may be too tedious to perform in a manual fashion, in which case there are two other choices:
The policy server trace log can be used to determine the Realm to which the request mapped (be sure that Thread ID (tid) is part of the policy server trace config format as this helps greatly with separating individual requests from the log). The Realm can then be examined in the Admin UI.
The third way the Realm can be determined is by using the SM Test Tool (Windows only). You will need the resolved AgentName and URI from the Web Agent trace log. When you click the IsProtected button in the Test Tool, the Realm to which the request resolved will be displayed. The Realm can then be examined in the Admin UI.