NTLM Authentication Fails with 401 Unauthorized via NSX Load Balancer for MSIE6 User-Agents

Products

• VMware NSX

Issue/Introduction

• An NTLM authentication issue occurs when accessing web services published through an NSX Load Balancer Virtual IP (VIP) 
• The authentication negotiation succeeds or fails based entirely on the client's User-Agent header:
  • Successful Request Example:
    • User-Agent: Mozilla/5.0 (Windows NT; Windows NT 10.0) WindowsPowerShell/5.1
    • Result: HTTP 200 OK
    • Response Headers: Connection: keep-alive, Persistent-Auth: true
  • Failed Request Example (ASMX/.NET Client):
    • User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol)
    • Result: HTTP 401 Unauthorized
    • Response Headers: Connection: close, WWW-Authenticate: NTLM
• The failure occurs because the TCP connection is closed prematurely during the NTLM challenge instead of being kept alive to complete the authentication sequence.

Environment

• Vmware NSX Load Balancer
• MSIE 6.0

Cause

The native NSX Load Balancer is built upon an NGINX proxy engine. By default, NGINX implements the keepalive_disable msie6; directive (Module ngx_http_core_module). This hardcoded setting forcefully drops TCP keep-alive connections after an initial POST response is sent to any client identifying as MSIE 6.0. Dropping the connection breaks multi-stage authentication protocols (such as NTLM) which require the TCP socket to remain completely persistent across the 401 challenge and subsequent authorization requests.

Resolution

To bypass the legacy NGINX engine limitation, implement one of the following options:

1. Modify Client Configuration (Workaround): Update the client application stack or codebase to send a newer browser emulation identifier in the User-Agent string (e.g., MSIE 7.0 or higher). This prevents the load balancer from triggering the keep-alive drop.

2. Migrate to NSX Advanced Load Balancer (Avi): Migrate the affected application to the NSX Advanced Load Balancer (Avi). Avi's proprietary Service Engine proxy does not utilize NGINX and it has an option to allow MSIE6 connection as per Keep-alive Connection Hung Issue for MSIE 6.0.