Gorouter returns 502 error when apps returns http status 200 with invalid content length

Products

• VMware Tanzu Application Service

Issue/Introduction

After upgrading TAS to 4.0.30+, 6.0.10+, or 10.0.0+ Gorouter access.log or app logs may report an error in the x_cf_routeerror field which states a bad content-length header.  Upon network inspection we will observe the app sending a HTTP 1.0 or 1.1 response with status code 200.  Gorouter reports the response as invalid and returns an http status 502 error.

 

[RTR/2] [OUT] app.domain - [2025-01-30T19:41:02.052635461Z] "GET / HTTP/2.0" 502 0 67 ... "x_cf_routererror:"endpoint_failure (net/http: HTTP/1.x transport connection broken: bad Content-Length \"[13411]\")" ...

Cause

The gorouter returns this error if the application returns a response that does not have a valid content-length header.  

 

There was a change in GO 1.23 ( commit ) which attempts to parse a content length header when a transfer encoding header is set.  

 

Before GO 1.23 the go reverse proxy would simply ignore the content length header and allow invalid “Content-Length” headers to pass through.  According to the commit in go 1.23 invalid "Content-Length" headers might involve request smuggling or response splitting, which is a security concern. Going forward these headers will be checked for validity even when transfer encoding headers are set.

Here is an example response from the app which will trigger the gorotuer to return a 502. In this example the Content-Length header includes brackets “[“ and “]” which are non-numeric characters.  RFC 2616 expects the value of content length to be a numeric value only. Prior to TAS 4.0.30 and 6.0.10 this invalid content length header would have been ignored when the transfer encoding header is set and the request will pass through the gorouter successfully. 

HTTP/1.1 200 
Content-Length: [13411]
X-Content-Type-Options: [nosniff]
Content-Encoding: gzip
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked

 

Currently If the app response had a valid content length, which is equal to the size of the encoded body, then gorouter will not return an error.  In the following example removing the brackets “[“ “]” in the content-length header is an acceptable response. Although it is not recommended to include a content-length header when transfer encoding is set and ideally for security reasons the response should not include it anyway.

HTTP/1.1 200 
Content-Length: 13411
X-Content-Type-Options: [nosniff]
Content-Encoding: gzip
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked

Resolution

Routing Release 0.312.0 and later now builds with go 1.23. This behavior is expected and will be observed in TPCF 4.0.30+, 6.0.10+, and 10.0.  

The application in question should be updated to not include a content-length header when transfer encoding is set.  

See links below for more information explaining how this is not a bug and why it is important to ensure your application is updated to meet the industry specification for http.

 

• https://www.rfc-editor.org/rfc/rfc9112.html#section-6.1-12
• Explains how allowing invalid content-length headers is a security concern
• https://www.rfc-editor.org/rfc/rfc9110.html#name-content-length
• Explains content-length semantics