All HTTP/2 requests fail/hang with no obvious indicator pointing to :scheme

Products

• VMware Tanzu Spring Essentials
• Support Only for Apache Tomcat
• VMware Tanzu tc Server

Issue/Introduction

10.1.55 added (changelog, Coyote section):

 "Add validation that the HTTP/2 :scheme pseudo-header is consistent with the use (or not) of TLS. (markt)"

Implementation in java/org/apache/coyote/http2/Stream.java (10.1.55, ~line 551):

    if ("https".equals(value) != handler.getProtocol().getHttp11Protocol().isSSLEnabled()) {
        headerException = new StreamException(
            sm.getString("stream.header.inconsistentScheme", getConnectionId(), getIdAsString(),
                value, Boolean.toString(handler.getProtocol().getHttp11Protocol().isSSLEnabled())),
            Http2Error.PROTOCOL_ERROR, getIdAsInt());
    }

The check raises PROTOCOL_ERROR and resets the stream when the inbound :scheme does not match the connector's SSLEnabled flag.

This breaks a very common deployment topology: a TLS-terminating reverse proxy that forwards HTTP/2 cleartext (h2c) to the backend while preserving the original request's :scheme: https. Concretely, this is what Cloud Foundry's GoRouter does when a route destination has protocol: http2 -- the public hop is HTTPS, GoRouter terminates TLS, and forwards as h2c to the container with :scheme: https. The same pattern applies to Envoy, nginx, HAProxy, and anything else doing edge-TLS + h2c upstream.

In our environment (SAP Java Buildpack 2.61.0, which bundled Tomcat 10.1.55) every HTTP/2 request via GoRouter started failing immediately on upgrade from 10.1.54. Bisected to tomcat-coyote.jar 10.1.54 -> 10.1.55. Direct in-container reproducer below.

 

REPRODUCER (MINIMAL)
====================

Plain Tomcat 10.1.55, default <Connector> with <UpgradeProtocol Http2Protocol/>,
no TLS:

    curl --http2-prior-knowledge -H ':scheme: https' http://127.0.0.1:8080/
    # curl: (55) Failed sending HTTP request

Same against 10.1.54 -> 200 OK.

End-to-end on Cloud Foundry: deploy any Java app via SJB 2.61 (Tomcat 10.1.55), set the route destination to protocol: http2 (cf curl /v3/routes/<guid>/destinations -X PATCH ...), then curl https://<route>/ from outside. Hangs / endpoint_failure (http2: client conn could not be established) until upstream timeout. Switching the destination back to protocol: http1 makes everything work, with 10.1.55 still in place -- confirming the trigger is the :scheme: https over plaintext-TCP combination, not anything else.

Environment

• Tomcat 10.1.55 (via SJB 2.61 or Java Buildpack 4.93.0)
• Reverse proxy: Cloud Foundry GoRouter (h2 frontend, h2c upstream when destination protocol: http2)
  • A TLS-terminating reverse proxy forwarding h2c to the backend connector
• Connector: plain (non-TLS) HTTP/1.1 with <UpgradeProtocol Http2Protocol/>
• Last working version: Tomcat 10.1.54 (same JRE, same proxy, same app, same connector config)

Resolution

Implemented a new option for Http2Protocol called "allowSchemeMismatch" that defaults to "false" but can be set to "true" when Tomcat is behind a proxy that is trusted to set the HTTP/2 scheme correctly.

Fixed in:

- 11.0.x for 11.0.23 onwards
- 10.1.x for 10.1.56 onwards
-  9.0.x for  9.0.119 onwards

Additional Information

Bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=70091