Tanzu Build Service images cannot be pulled from Harbor running on Tanzu Kubernetes Grid 1.2.x

Products

• VMware Tanzu Build Service
• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

Symptoms:

• Tanzu Build Service images cannot be pulled from Harbor running on Tanzu Kubernetes Grid 1.2.x nodes
• Harbor is deployed as a shared service and is using vSphere CNS/CSI for the backing storage
• Contour, deployed as an extension, is used for the ingress to Harbor
• You see a message similar to "failed to copy: unexpected EOF"
• You see a message in the harbor-core pod log that says: "Handler crashed with error net/http: abort Handler"

Environment

VMware Tanzu Kubernetes Grid 1.x
VMware Tanzu Kubernetes Grid Plus 1.x

Resolution

This is a known issue affecting Harbor and Contour on Tanzu Kubernetes Grid 1.2.x. There is currently no resolution.

Workaround:
To workaround this issue, you can increase the Contour response timeout value from the default of 15 seconds to a higher value.

1. Open the tkg-extensions-v1.2.0+vmware.1/extensions/registry/harbor/harbor-extension.yaml file with a text editor.
2. Update the spec -> objects -> spec -> template section such that it looks like the following:

      template:
        - ytt:
            ignoreUnknownComments: true
            paths:
              - tkg-extensions/common
              - tkg-extensions/registry/harbor
            inline:
              pathsFrom:
                - secretRef:
                    name: harbor-data-values
              paths:
                update-harbor-http-proxy.yaml: |
                  #@ load("@ytt:overlay", "overlay")
                  #@ harbor_httpproxy = overlay.subset({"kind": "HTTPProxy", "metadata": {"name": "harbor-httpproxy"}})
                  #@overlay/match by=harbor_httpproxy
                  ---
                  spec:
                    routes:
                      - #@overlay/match by=overlay.index(3)
                        #@overlay/match missing_ok=True
                        timeoutPolicy:
                          response: 60s

Note: The paths section is added to the original file, with the response value set to 60s.

3. Save and close the file.
4. Apply the changes made to the file by running the kubectl apply -f tkg-extensions-v1.2.0+vmware.1/extensions/registry/harbor/harbor-extension.yaml command.

Note: You can validate that the change has been made by running the following command:

kubectl get httpproxy -n tanzu-system-registry  harbor-httpproxy -o yaml | grep 'prefix: /v2/' -A 5 -B 1

You should see the following output:

 - conditions:
   - prefix: /v2/
   services:
   - name: harbor-core
     port: 443
   timeoutPolicy:
     response: 60s


Alternatively, this issue has not been observed when using S3 storage. You can reconfigure Harbor to use an S3 storage location instead of vSphere CNS/CSI.