"too many transfer encodings" 502 chunked response errors when running applications on Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms

Users of applications running on Tanzu Application Service (TAS) for VMs may experience unexpected 502 errors. Upon looking into /var/vcap/sys/log/gorouter/gorouter.stdout.log, you will see "too many transfer encodings" in the error message. This happens after upgrading to TAS for VMs or Isolation Segment.

Routing Release 0.209.0+

• TAS for VMs
  • 2.7.30
  • 2.8.24
  • 2.9.18
  • 2.10.10
• Isolation Segments 
  • 2.7.29
  • 2.8.23
  • 2.9.17
  • 2.10.9

A recent issue was discovered in later releases of TAS for VMs where some versions of TAS will silently remove the Transfer encoding header and replace it with a content length header. This leads to a false sense of mitigation and we recommend reviewing this related issue. For more information, refer to Clients receive responses with no Content-Length header and a chunked encoded body after upgrading Tanzu Application Service for VMs.

The following is an example of the error message you receive:

"error":"net/http: HTTP/1.x transport connection broken: too many transfer encodings: 
[\"chunked\" \"chunked\"]

 

Why this is happening

An application may add "Transfer-Encoding: chunked" or other transfer encoding headers. However, the need to do this is rare as most often your web server, a library, or a framework is going to do this automatically, as necessary, for you. The error described in this KB occurs if any response from an application has multiple transfer-encoding headers, either directly adding or a framework adds on its behalf, or a single transfer-encoding header with a value other than "chunked". For the case of multiple headers, it does not matter if there are two identical headers with the same value or if there are two transfer-encoding headers with different values.

The reason the error listed above is happening in the specific releases mentioned above is that starting with these releases Gorouter has been built with Golang 1.15. Before Golang 1.15, duplicated transfer-encoding headers or headers with two values were permitted by Golang's proxy roundTrip. However, starting in Golang 1.15, the implementation is more strict. Specifically, the router is now rejecting responses from applications with either multiple transfer encodings or out-of-spec "identity" encoding values. Golang 1.15 is more strict about the Transfer Encoding header to prevent a well-known request smuggling vulnerability, as well as mitigate a broad class of security issues around this header over time.

Triggers

This can happen under a number of scenarios. The following is a list of the scenarios that VMware Tanzu Support has specifically observed causing this error:

1. An application proxies requests (like a gateway) and the proxy/gateway app is naïvely copying all headers between the client and backend application. This can cause issues because you may copy a transfer encoding header added by the server to which you're communicating and also have the proxy server add in a second transfer-encoding header, which can trigger the error. It is incorrect for a proxy/gateway application to copy the transfer-encoding header, which is a hop-by-hop header.
2. When creating a route service. A route service works similar to a proxy, but instead of relaying requests, it is inspecting requests and possibly augmenting them. If the route service causes a second transfer-encoding header to be added, this will trigger the error.
3. In Spring-based microservice applications, when using RestTemplate from a Controller annotated class and returning the result of RestTemplate.exchange in a method that returns a ResponseEntity. Similar to situation #1, this will cause a problem if the microservice to which you are talking returns a transfer-encoding header because RestTemplate.exchange will copy all of the response headers from the client response into the ResponseEntity it creates, including the transfer-encoding header.
4. An application author is specifically adding a transfer-encoding header, unaware that their server, a library, or a framework is already handling this on their behalf.

Summary

1. TAS for VMs  2.7.30, 2.8.24, 2.9.18 or 2.10.10, or Isolation Segment 2.7.29, 2.8.23, 2.9.17, 2.10.9 all now compile Gorouter using Golang 1.15.
2. Golang 1.15 introduces a breaking change that rejects HTTP requests that have a Transfer-Encoding header of value other than "chunked" or more than one Transfer-Encoding header (i.e. “chunked” twice).
3. Transfer-Encoding headers with the value "identity" were originally a synonym for not passing this header at all. The HTTP/1.1 spec removed this value in a revision RFC 7230, and Golang removed support for it in Golang 1.15.
4. Golang 1.14 and prior never supported transfer encodings other than chunked and identity, such as deflate or gzip, but it was less strict about the use of chunked headers. Golang is attempting to resolve a variety of security issues (including a very dangerous request smuggling vulnerability) around the Transfer-Encoding header by being very strict in what it allows.

Environment

Product Version: 2.9

Resolution

Detecting or Monitoring For This Issue

Detecting and monitoring this issue is complex and requires an upgrade to TAS versions 2.7.31, 2.8.25, 2.9.19, and 2.10.11, or Isolation Segment 2.7.30, 2.8.24, 2.9.18, 2.10.10. This process has been further explained and in the KB titled How to detect applications that return multiple transfer-encoding headers.

Code Review/Fix

In general, to resolve this issue you need to review applications experiencing these errors and make code changes to ensure that multiple or duplicate transfer-encoding headers are not being returned to Gorouter. This is not a change or fix that can be made on the platform itself.

VMware Tanzu Support cannot review your code or provide specific code fixes, however, for Spring based applications we can offer the following advise to not trigger this issue.

1. For streaming results from the server to a client, use Spring's built-in support for this. Using a ResponseBodyEmitter or a SseEmitter, you can easily stream content back to your clients and Spring & Tomcat will ensure that the transfer-encoding header is set correctly.

2. If you created a route service based on this example code, https://github.com/nebhale/route-service-example (no longer published due to this bug), you will be impacted as the example code was copying all response headers, include transfer-encoding headers from the proxied response to the client response (see the bold line below).

           return this.webClient
            .method(request.getMethod())
            .uri(forwardedUrl)
            .headers(headers -> headers.putAll(forwardedHttpHeaders))
            .body((outputMessage, context) -> outputMessage.writeWith(request.getBody()))
            .exchange()
            .map(response -> {
                this.logger.info("Outgoing Response: {}", formatResponse(response.statusCode(), response.headers().asHttpHeaders()));

                return ResponseEntity
                    .status(response.statusCode())
                    .headers(response.headers().asHttpHeaders())
                    .body(response.bodyToFlux(DataBuffer.class));
            });

   To implement correctly, you will need to strip out the transfer-encoding header, like in this example.

   return this.webClient
            .method(request.getMethod())
            .uri(forwardedUrl)
            .headers(headers -> headers.putAll(forwardedHttpHeaders))
            .body((outputMessage, context) -> outputMessage.writeWith(request.getBody()))
            .exchange()
            .map(response -> {
                HttpHeaders headers = getResponseHeaders(response.headers().asHttpHeaders());

                this.logger.info("Outgoing Response: {}", formatResponse(response.statusCode(), headers));

                return ResponseEntity
                    .status(response.statusCode())
                    .headers(headers)
                    .body(response.bodyToFlux(DataBuffer.class));
            });
    }

   Where the getResponseHeaders function looks like this:

   private HttpHeaders getResponseHeaders(HttpHeaders headers) {
    return headers.entrySet().stream()
        .filter(entry -> !entry.getKey().equalsIgnoreCase(TRANSFER_ENCODING))
        .collect(HttpHeaders::new, (httpHeaders, entry) -> httpHeaders.addAll(entry.getKey(), entry.getValue()), HttpHeaders::putAll);
}

   As an additional note beyond the context of this issue, it is not good practice to copy every header when proxying traffic. The example above strips out a few request headers and the transfer-encoding header on the response, but you would want to be more restrictive for production applications to prohibit clients from sending headers to manipulate backend applications in ways they should not be able to do (like the HTTPoxy vulnerability) and to prohibit information, like CORS headers, from leaking out of backend services to clients.

3. Situation #3 above, can be mitigated by not directly returning ResponseEntity objects from RestTemplate.exchange. You need to first remove the transfer-encoding header, if it's present. You could directly modify that object in your Controller before returning it, but that could get repetitive across many methods and many controllers. A less invasive way of doing this would be with a RestTemplate interceptor.
4. Another option is to utilize Spring WebClient. You may use WebClient in both Spring MVC (Servlet) and Spring Webflux apps. With WebClient, you can return a Flux and that will trigger a streamed response (i.e. transfer-encoding chunked). In this situation, the transfer-encoding header will only be set once.
5. If you are manually adding any Transfer-Encoding headers, remove them. How you add headers will vary from one language/framework to another. An example in Java would be using HttpServletResponse.addHeader to add a transfer-encoding header.

Rollback

While not recommended, a short-term workaround is to roll back to an older version of routing-release that uses Golang 1.14. This is only a temporary solution for two reasons.

1. Golang 1.14 will only be supported until February 1, 2021, when Golang 1.16 is released. All fixes that go into the Golang runtime after February 1, 2021 will happen in newer releases which will include the stricter implementation that rejects multiple/duplicate transfer-encoding headers.
2. All future VMware Tanzu Application Service updates will be built using Golang 1.15+, thus you will not be able to update your foundation until these application issues have been resolved.