Error "upstream connect error or disconnect/reset before headers" for http routes in a Global Namespace (GNS) in Tanzu Service Mesh

Products

• VMware Tanzu Service Mesh

Issue/Introduction

Connecting to a public service with a HTTP route in a GNS can fail with a "HTTP/2 503" code and the error message:

upstream connect error or disconnect/reset before headers

 

However, when the HTTP/1.1 protocol is used (by using the following curl command as an example), then it connects successfully with a "200" code.

curl https://servicename.poc1.tsm -vvv --http1.1

Environment

Tanzu Service Mesh

Cause

In TSM service, the "useClientProtocol" setting has been enabled by default and is causing the issue unintentionally.  Disabling the setting globally in TSM is not an option at this time as it might cause further problems.

Resolution

As a workaround:

Overwrite the "useClientProtocol" setting in the "DestinationRule" for the GNS by creating another "DestinationRule" (a copy of existing "DestinationRule") for specific service with "useClientProtocol" set to "false".  

In the new "DestinationRule", the "host" must be set to specific service (i.e., instead of "*.poc1.tsm" use "servicename.poc1.tsm"), so that the GNS "DestinationRule" is overwritten for the particular service by that new "DestinationRule".  See the example below.

 

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: nsxsm.gns.poc1.tsm
  namespace: istio-system
spec:
  host: 'servicename.poc1.tsm'
  trafficPolicy:
    connectionPool:
      http:
        useClientProtocol: false
    loadBalancer:
      localityLbSetting:
        enabled: false
      simple: ROUND_ROBIN
    outlierDetection:
      baseEjectionTime: 120s
      consecutive5xxErrors: 1
      consecutiveGatewayErrors: 1
      interval: 1s
      maxEjectionPercent: 50
      minHealthPercent: 1
    tls:
      mode: ISTIO_MUTUAL