Customer noticed that NTLM authentication was failing when hitting specific BCAAA servers but working fine when the authentication requests was hitting a different BCAAA servers.
Kerberos authentication worked fine via both BCAAA servers.
PCAP simply showed that the proxy was returning a HTTP 500 internal server error to the client
ProxySG eventlog showed a generic message
2016-09-13 13:30:25+01:00BST "Unrecognised error reported to authentication agent." 2D 3B0003:1 pe_policy_action_auth_internal.cpp:676
BCAAA windows eventlog was showing
6887.303 [email protected][IWA_Realm]: Error returned from NTLM agent: 0x250129
Enabling BCAAA debug logs (see How do I enable BCAAA debug logging?) showed that BCAAA was returning the following error
[15520:21700] AcceptSecurityContext failure, ContextLink=0x0 count=0, detail=1(Incorrect function.); status=-2146893054:0x80090302:The function requested is not supported
Based on the BCAAA debug error messages the issue pointed to an incompatibility in NTLM security settings between the client and the BCAAA server. More specifically the value of NtlmMinClientSec in the BCAAA servers registry. (See How to enable NTLM 2 authentication for some background information on this setting)
Checking the NtlmMinClientSec registry entry showed the value to be 0x20080000 setting it back to the defualt value of 0x20000000got NTLM authentication working again