Clients unable to access application behind NSX-T Load Balancer after NSX upgrade

Products

• VMware NSX

Issue/Introduction

This issue pertains to Layer 7 NSX-T native load balancers

In the Edge syslog, the following error may be observed. Note: \x20 is the hexadecimal value for the ASCII space character.

2025-06-23T13:36:06.516Z example.abc.com NSX 1865577 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="ERROR" errorCode="EDG9999999"] [######-####-####-####-#########] upstream sent invalid header: "\x20..." while reading response header from upstream, client: 192.168.1.1, server: , request: "GET /TEST HTTP/1.1", upstream: "http://1.1.1.1:80/TEST", host: "abc.example.com"

Environment

NSX-T 3.2.2 and later

Cause

A packet capture on the pool member's (application server) switch port shows an extra space in the HTTP header. This is flagged by Wireshark as:
"Illegal character found in header name." See reference screenshot from Wireshark below 

This behavior is due to a change in Nginx. Starting with Nginx 1.21.1, control characters (0x00–0x1F, 0x7F), spaces, and colons are no longer permitted in HTTP header names and are explicitly rejected.

In NSX-T version 3.2.2, the bundled Nginx was upgraded from 1.18.0 to 1.22.0, leading to this issue being observed from version 3.2.2 onward.

Resolution

Update the backend application code to ensure that HTTP headers do not contain invalid characters (0x00–0x1F, 0x7F), spaces, and colons, to comply with the updated Nginx behavior

Workaround

If modifying the backend application is not feasible, consider switching to Layer 4 load balancing