NTLM authentication fails with HTTP 500 internal server error

Products

• ProxySG Software - SGOS

Issue/Introduction

The customer noticed that NTLM authentication was failing when hitting specific BCAAA servers but working fine when the authentication requests were hitting a different BCAAA server.

Kerberos authentication worked fine via both BCAAA servers.

PCAP simply showed that the proxy was returning a HTTP 500 internal server error to the client

ProxySG eventlog showed a generic message
2016-09-13 13:30:25+01:00BST  "Unrecognised error reported to authentication agent."  2D 3B0003:1   pe_policy_action_auth_internal.cpp:676

BCAAA windows eventlog was showing 
6887.303 NTLMAuthenticateRCB@0x1F97E784F0[IWA_Realm]: Error returned from NTLM agent: 0x250129 

Enabling BCAAA debug logs (see How do I enable BCAAA debug logging? : https://knowledge.broadcom.com/external/article/166076/gather-bcaaa-debug-logs.html) showed that BCAAA was returning the following error
[15520:21700] AcceptSecurityContext failure, ContextLink=0x0 count=0, detail=1(Incorrect function.); status=-2146893054:0x80090302:The function requested is not supported 

 

Cause

Based on the BCAAA debug error messages the issue pointed to an incompatibility in NTLM security settings between the client and the BCAAA server. More specifically the value of NtlmMinClientSec in the BCAAA servers registry. (See  How to enable NTLM 2 authentication for some background information on this setting)

Resolution

Checking the NtlmMinClientSec registry entry showed the value to be 0x20080000 setting it back to the defualt value of 0x20000000got NTLM authentication working again