Unexpected 400 responses from RFC non-compliant app servers if gorouter is built with golang v1.20+

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

In golang version 1.20 and beyond, there's a significant update that affects how gorouter manages requests with the Expect: 100-continue header. This change can sometimes result in unexpected 400 responses from RFC non-compliant app servers running on Tanzu Application Service for VMs (TAS) in certain situations. The purpose of this modification in Golang is to improve compliance with RFC7231 Section 6.2.

The following Tanzu products contain a routing release that is compiled with golang v1.20+:

Tanzu Application Service for VMs 

• 2.11.36+
• 2.13.18+
• 3.0.8+
• 4.0.0+

Tanzu Isolation Segment releases

• 2.11.30+
• 2.13.15+
• 3.0.8+
• 4.0.0+

TAS Clients and Servers that do not follow RFC standards might be affected by this change, as gorouter now treats requests with the Expect: 100-continue header differently. 

This Knowledge Base (KB) article is specific to the server impact. To see more details on the client impact, please see this KB article.

There are a few concepts to cover in order to understand how this change affects TAS app servers.

Purpose of the Expect: 100-continue header
Think of the Expect: 100-continue header as a way to prevent sending large files to a server that might not be ready for them. For instance, if an app server can accept a 500mb file from authorized clients, using this header allows the server to check the request before receiving the entire 500mb file. If the request is found to be invalid, like missing headers, the server stops the client from sending the large file, which saves data and makes the process more efficient.

To see how the Expect: 100-continue header looks on the wire for a successful request consider the following curl command and app server network trace:

curl -X POST exp####-####-#####-##.#####-##.####-##.####.####.com/ --header 'Expect: 100-continue'  -d 'mydata910'

POST / HTTP/1.1
Host: #########-######-#######-##.#####-##.####-20.###.#####.com
User-Agent: ####/8.#.#
Content-Length: 9
Expect: 100-continue
<omitted for brevity>

HTTP/1.1 100 Continue

my#####910

HTTP/1.1 200 OK
transfer-encoding: chunked
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
<omitted for brevity>

Notice how the POST body payload "my####910" was not transmitted by the client until the server responded with a "100 continue". 


TAS gorouter keep-alive logic
Gorouter has support for keep-alive connections with back end. TAS enables this option by default and the keep-alive timeout is 90 seconds. To grasp the concept, let's take an example: imagine there is an app called AppA, and it only runs one instance:

• When Gorouter gets the initial request for AppA, it sets up a connection to that specific backend instance. After AppA responds, Gorouter sends this response back to the client. If the response doesn't signal the server's intent to close the connection, Gorouter maintains this connection between itself and AppA for 90 seconds.
• If Gorouter receives another request for AppA within these 90 seconds, it will reuse the existing connection instead of creating a new one. However, if there are two concurrent requests for AppA and the first request isn't finished when Gorouter starts the second one, it will establish a new connection to the backend instance. Once both requests are done, Gorouter will have two connections available for future requests to AppA within the 90-second connection timeout.
• On a per-connection basis, if 90 seconds pass without another request for AppA, Gorouter will close the connection to that backend instance, and this cycle repeats.

Gorouter behavior changes now that it is compiled with golang v1.20+
Prior to v1.20, golang’s proxy logic sent an immediate 100-continue to clients in response to the Expect: 100-continue header to encourage clients to send data immediately. When the backend server would send its own 100-continue, the proxy logic would swallow that duplicate 100-continue message. Starting in v1.20, golang's proxy logic stopped swallowing that message. Due to the potential of duplicate 100-continue  responses breaking RFC non-compliant clients, gorouter implemented a change to delay sending its 100-continue  by one second in an effort to see if the server sends the 100-continue response first. If the server does send the 100-continue response within one second then gorouter will not generate its own 100-continue. For more insight into this see the "nitty gritty" details section in this routing release doc.


Once the above concepts are clear we can proceed to understanding the impact that RFC non-compliant app servers face on TAS. At the beginning of this KB article it was mentioned these servers could face "unexpected 400 responses in certain situations" The certain situation is as follows:

• Connection reuse 
• Request-1 contains an Expect: 100-continue header and a content-length > 0
• The server responds to Request-1 with a non 100/200 status code before reading the body content or closing the connection.
• Request-2 on the same connection is immediately seen as malformed and response code 400 is returned to client.

Lets see how this looks in a network trace where a single connection handles two requests:

#Request-1
POST / HTTP/1.1
Host: ########-######-######-##.#######-##.###-##.###.####.com
User-Agent: #####/8.#.2
Content-Length: 9
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue
<omitted for brevity>

HTTP/1.1 401 Unauthorized
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
<omitted for brevity>

#Request-2
GET / HTTP/1.1
Host: ########-#######-#########-##.######-##.#####-##.###.#####.com
User-Agent: #####/8.#.2
<omitted for brevity>

HTTP/1.1 400 Bad Request
content-length: 0
connection: close

In Request-1, there's a POST request with a 9-byte payload and the Expect: 100-continue header. The server checked this request but returned a 401 unauthorized response for some reason. Because the server doesn't close the connection and didn't read Request-1's content body, it expects the next incoming bytes to belong to Request-1's payload. When it instead receives Request-2's bytes, it sees this as a problem and immediately responds with a 400 error and closes the connection.

Environment

Product Version: 2.11

Resolution

Per RFC
"A server that responds with a final status code before reading the entire request content SHOULD indicate whether it intends to close the connection (e.g., see Section 9.6 of [HTTP/1.1]) or continue reading the request content."

Many servers account for this scenario, for example tomcat and go http servers will close the connection on any non 100/200 response code requests that had expectations (such as the Expect: 100-continue header). Here is how it looks on the wire for a RFC compliant server which receives a request with the Expect: 100-continue header but returns an immediate 401 before reading the body:

POST /hello HTTP/1.1
Host: mygoapp-wise-mandrill-ki.cfapps-31.slot-35.#####-####-####.######.com
User-Agent: curl/8.1.2
Content-Length: 9
Expect: 100-continue
<omitted for brevity>

HTTP/1.1 401 Unauthorized
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Thu, 21 Sep 2023 20:02:43 GMT
Content-Length: 13
Connection: close

Unauthorized

The go http server closed this connection automatically thus it is RFC compliant.

Because of the way the Go proxy in Gorouter is designed at a low level, it's not practical to create a patch in Gorouter to accommodate servers that don't follow RFC standards. Instead, the recommendation is to address this issue in the server itself by either reading the content body or closing the connection. Ideally, these edge cases should be handled at a lower abstract level, like within the app's server itself. An example of this approach can be observed in the following GitHub issue in the Reactor-Netty project. The patch for Reactor Netty server v1.0.37 will be available in Spring Framework 5.x+ and Spring Boot 2.7.x+.

If a situation arises where there's a need to fix a server application lacking RFC compliance, and no server version is available to handle this at the abstract layers, a workaround can be programmatically implemented within the application code itself. This workaround involves reading the body content before responding with the final status code or closing the connection with the response, and it serves as a temporary solution until an automatically handling server version becomes available.


Being Proactive
It is possible to identify applications who have clients leveraging the Expect: 100-continue header by making gorouter log this header in its access logs. In the TAS tile -> Networking section this can be enabled:

Then the RTR/access log will include that header:

mygoapp-grateful-aardvark-sp.cfapps-37.slot-34.###-###-###.####.com - [2023-09-28T18:00:51.559351236Z] "POST /hello HTTP/1.1" 200 9 23 "-" "curl/8.1.2" "10.###.##.##:54442" "10.###.##.##:61050" x_forwarded_for:"10.##.###.84, 10.###.##.60" x_forwarded_proto:"http" vcap_request_id:"9251ccc0-####-####-####-678e800eafd5" response_time:0.126257 gorouter_time:0.001333 app_id:"#######-4eab-476d-######-df23f075460b" app_index:"0" instance_id:"e1ca8756-7b89-####-50d2-####" x_cf_routererror:"-" expect:"100-continue" x_b3_traceid:"45d###########1eda50e8829798" x_b3_spanid:"451e########3829798" x_b3_parentspanid:"-" b3:"45d672903##############829798-451eda50e8829798"