Harbor replication timeout - "context deadline exceeded (Client.Timeout or context cancellation while reading body)"

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

A large number of Harbor replication jobs remained at the "InProgress" status from the Harbor WebUI (Administration --> Replications --> Executions).

The harbor-jobservice pod log showed below error message.

kubectl -n tanzu-system-registry logs deployment.apps/harbor-jobservice | grep ERROR
#> YYYY-MM-DDThh:mm:ssZ [ERROR] [/controller/replication/transfer/image/transfer.go:395]: failed to pushing the blob sha256:############, size 2455630552: Put "https://harbor-core:443/v2/staging-kkc/kkc_app_portal_0fl/blobs/uploads/#############?_state=###########": context deadline exceeded (Client.Timeout or context cancellation while reading body)

Environment

Harbor v2.9.1 from Harbor Standard Package

Cause

The default replication timeout value in Harbor Registry is 30 minutes. Replicating large container images (10–20 GB) could exceed this timeout and fail.
Upstream ref. - goharbor - Replication timeout between two harbors

Resolution

Apply REGISTRY_HTTP_CLIENT_TIMEOUT for harbor-jobservice by using YTT overlay.

1. Make the Registry Read Only mode

Harbor WebUI --> Administration --> Configuration --> System Settings --> Check "Repository Read Only" --> SAVE
Reference - Harbor Official Document

2. Generate a YTT overlay YAML file - Set the REGISTRY_HTTP_CLIENT_TIMEOUT to 60 minutes

cat > pkg-harbor-overlay.yaml <<EOF
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({"kind": "Deployment", "metadata": {"name": "harbor-jobservice"}})
---
spec: 
  template: 
    spec: 
      containers: 
        #@overlay/match by=overlay.subset({"name": "jobservice"})
        - env: 
          #@overlay/add
          - name: REGISTRY_HTTP_CLIENT_TIMEOUT
            value: "60"
EOF

3. Update the harbor package

kubectl get pkgi -A | grep -E 'NAME|harbor'
#> NAMESPACE    NAME      PACKAGE NAME               PACKAGE VERSION         DESCRIPTION           AGE
#> mypackage    harbor    harbor.tanzu.vmware.com    2.9.1+vmware.1-tkg.1    Reconcile succeeded   6d22h
pkg_namespace=mypackage

# Update the package
tanzu -n ${pkg_namespace} package installed update harbor --ytt-overlay-file pkg-harbor-overlay.yaml
#> ...
#> hh:mm:ss: Deploy succeeded

4. Verify the result

## Check - all Harbor pods are at "Running" state
kubectl -n tanzu-system-registry get all

## Check - Deployment
kubectl -n tanzu-system-registry get deploy harbor-jobservice -oyaml | grep -A1 'name: REGISTRY_HTTP_CLIENT_TIMEOUT'
    - name: REGISTRY_HTTP_CLIENT_TIMEOUT
      value: "60"

## Check - Pod
pod=$(kubectl -n tanzu-system-registry get pod -l app=harbor,component=jobservice -o jsonpath='{.items[0].metadata.name}')
kubectl -n tanzu-system-registry exec $pod -- env | grep REGISTRY_HTTP_CLIENT_TIMEOUT
REGISTRY_HTTP_CLIENT_TIMEOUT=60

5. Revert the Harbor registry READONLY mode

Harbor WebUI --> Administration --> Configuration --> System Settings --> Uncheck "Repository Read Only" --> SAVE

Additional Information

You can check the current YTT overlay YAML file contents using below command:

kubectl -n ${pkg_namespace} get secrets harbor-${pkg_namespace}-overlays -oyaml | yq '.data["0000-pkg-harbor-overlay.yaml"]' | base64 -d