Multiple HTTP Expect: 100-continue responses sent from gorouter to client may cause unexpected failures

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Recent Tanzu Application Service releases  2.11.36-2.11.41, 2.13.18-2.13.23, 3.0.8-3.0.13, and 4.0.0-4.0.4 and Tanzu Isolation Segment releases 2.11.30-2.11.35, 2.13.15-2.13.20, 3.0.8-3.0.13, and 4.0.0-4.0.4 include gorouter release 0.258.0-0.273.0 which introduces a change in how HTTP expect 100 headers are handled as a result of this update in Go 1.20.x. Go made this change in order to be compliant with RFC7231 Section-6.2.

Clients that are not compliant with RFC7231 may experience failures when the client sends an “Expect: 100-continue” header and then receives two HTTP 100 Continue responses, one from gorouter and one proxied through gorouter from the server.

Note: In this example curl is the client and it initiates a POST request including the “Expect: 100-continue” header. Gorouter returns two HTTP 100 Continue responses. In this example curl handles multiple 100 responses just fine, but some clients may exhibit unexpected behavior.

$ curl -v http://192.168.1.1:80 -H "Host: test-http.cfapps.domain.com" -H "Expect: 100-continue" --http1.1 -X POST -d 'hello'

< HTTP/1.1 100 Continue
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< X-Vcap-Request-Id: 63adec4c-7b10-44f1-4548-b2b3e8ca4810
< Date: Tue, 27 Jun 2023 15:05:30 GMT
< Connection: close
< Transfer-Encoding: chunked

Previously clients that sent a request with the header “Expect: 100-continue” only got one response back with status code 100 before getting their final response with the “real” status code. With go 1.20 they get two responses with status code 100. According to the HTTP 1.1 RFC, clients should be able to handle multiple responses with status code 100, however, some java spring clients may not handle this case and throw an exception. This can cause client apps to become unresponsive.   The faulty clients treat the second expect header as the final response with status code 100 and ignore the actual server's final status code of 200.
 

For more details about this bug and its behavior please review this routing release doc.

A second bug resulting from the go 1.20 changes is in the gorouter access log. You might observe an increase in HTTP 100 Continue responses logged in the gorouter access log that shows up in your reporting dashboards. We should expect gorouter to log the final status code of 200 instead of 100. This is a cosmetic bug and does not result in any impact to applications or clients.

test-http.cfapps.domain.com - [2023-06-26T16:50:54.486089317Z] "HEAD / HTTP/1.1" 100 0 0 "-" "curl/7.88.1" …<<truncated for brevity>>

Environment

Product Version: 4.0

Resolution

An upcoming TAS release will include routing release 0.274.0 which resolves both bugs mentioned in this KB. The table in this doc describes the fix behavior in detail.