Clients receive responses with no Content-Length header and a chunked encoded body after upgrading Tanzu Application Service for VMs
Article ID: 298288
Updated On:
Products
Issue/Introduction
After upgrading Tanzu Application Service for VMs (TAS for VMs), some clients may unexpectedly receive responses with no Content-Length header. Instead you receive responses with a chunked encoded body.
Affected versions
- TAS for VMs v2.11.3 or greater
- TAS for VMs v2.10.15 or greater
- TAS for VMs v2.9.23 or greater
- TAS for VMs v2.7.35 or greater
Why is this happening
Gorouter utilizes Golang’s proxy.go as the basis for its reverse proxy code. When compiled with Golang 1.15 or any prior version, a short response from an application with "Transfer-Encoding: chunked" is automatically "de-chunked" and returned by Gorouter with a Content-Length header instead.
Golang 1.16 now prioritizes flushing partial requests to the client and no longer changes the response to not be chunked.
Who is affected
If you have clients or Load Balancers that are unable to handle chunked encoding, they may have previously been relying on Gorouter to translate from chunked responses to non-chunked responses.
Gorouters compiled with Golang 1.16 now pass through the transfer encoding and respect the application's desire to chunk or not chunk the response from upstream applications.
Here are a couple of examples of users affected by this change in behavior:
- One user had a client that does not work with chunked responses properly after upgrading. They were unknowingly relying on Gorouter for converting chunked responses to non-chunked.
- The second user got incomplete responses, where 1 or more chunks are missing according to the off-platform app that's making API calls to an on-platform app. We suspect that the app response changed from automatic de-chunking to normal chunking. A component along the way is not waiting for all the chunks. We have not yet identified which component can not handle the chunked response.
Is this related to the "too many transfer encodings" errors
This issue is independent to "too many transfer encoding" errors. However, this issue does cause responses from applications running on the platform to have Transfer-Encoding headers when they previously did not. If there are proxies that are copying all headers, they may start to cause 502 errors when these behaviors are combined. In a previous upgrade Golang, and by proxy Gorouter, stopped accepting responses with multiple Transfer-Encoding headers. For more information about this behavior, refer to Applications on TAS for VMs get 502 chunked response error.
You might see "too many transfer encodings" in the error message. For example:
"error":"net/http: HTTP/1.x transport connection broken: too many transfer encodings: [\"chunked\" \"chunked\"]
This may have been triggered by an application now returning a chunked response to the proxy application that did not previously have a Transfer-Encoding header.
Why is this breaking change happening in point releases
Golang offers security patches for the current and previous versions of Golang. We try as much as possible to not change behavior in point releases. However, we prioritize security above maintaining identical behavior between releases. For this reason we will not be moving back to Golang 1.15.
Environment
Resolution
If you encountered this behavior after upgrading to Tanzu Application Service for VMs (TAS for VMs) v2.11.3 or higher, do one of the following to work around this issue:
- Fixing your clients to be able to handle a chunked response.
- As a temporary mitigation, applications can be modified to not send chunked responses. However, the HTTP/1.1 RFC states that “All HTTP/1.1 applications MUST be able to receive and decode the "chunked" transfer-coding”. Proxies, including Gorouter, may in the future change requests to be chunked.
Related code changes are documented in Applications on Tanzu Application Service for VMs get 502 chunked response error under Code Review/Fix.
Feedback
