Harbor replication timeout - "context deadline exceeded (Client.Timeout or context cancellation while reading body)"

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

A large number of Harbor replication jobs remained at the "InProgress" status from the Harbor WebUI (Administration --> Replications --> Executions).

The harbor-jobservice pod log showed below error message.

kubectl -n tanzu-system-registry logs deployment.apps/harbor-jobservice | grep ERROR
#> YYYY-MM-DDThh:mm:ssZ [ERROR] [/controller/replication/transfer/image/transfer.go:395]: failed to pushing the blob sha256:############, size 2455630552: Put "https://harbor-core:443/v2/staging-kkc/kkc_app_portal_0fl/blobs/uploads/#############?_state=###########": context deadline exceeded (Client.Timeout or context cancellation while reading body)

Environment

Harbor v2.9.1 from Harbor Standard Package

Habor 2.14.2 from Supervisor Services

Cause

The default replication timeout value in Harbor Registry is 30 minutes. Replicating large container images (10–20 GB) could exceed this timeout and fail.

Upstream ref. - goharbor - Replication timeout between two harbors

Resolution

Harbor Supervisor Services steps:

1. In the vSphere UI go to Hamburger Menu at the top left > Workload Management
   
   
   
   
2. In Workload Management, go to the services tab, find the harbor tile and click Actions > Manage Service
   
   
   
   
3. Select the supervisor where Harbor is deployed and click next
   
   
   
   
   

4. under jobservice before metric, add a new line that is the following: 

   REGISTRY_HTTP_CLIENT_TIMEOUT: 60
   
   
   
   

5. Select Finish, and it will apply the change

Harbor Standard Package steps:

Apply REGISTRY_HTTP_CLIENT_TIMEOUT for harbor-jobservice by using YTT overlay.

1.  Make the Registry READ ONLY mode
   
   Harbor WebUI --> Administration --> Configuration --> System Settings --> Check "Repository Read Only" --> SAVE
   
   Reference - Harbor Official Document
   
   
2. Generate a YTT overlay YAML file - Set the REGISTRY_HTTP_CLIENT_TIMEOUT to 60 minutes
   
   

   cat > pkg-harbor-overlay.yaml <<EOF
   #@ load("@ytt:overlay", "overlay")
   #@overlay/match by=overlay.subset({"kind": "Deployment", "metadata": {"name": "harbor-jobservice"}})
   ---
   spec: 
     template: 
       spec: 
         containers: 
           #@overlay/match by=overlay.subset({"name": "jobservice"})
           - env: 
             #@overlay/add
             - name: REGISTRY_HTTP_CLIENT_TIMEOUT
               value: "60"
   EOF

3. Update the harbor package
   
   

   kubectl get pkgi -A | grep -E 'NAME|harbor'
   #> NAMESPACE    NAME      PACKAGE NAME               PACKAGE VERSION         DESCRIPTION           AGE
   #> mypackage    harbor    harbor.tanzu.vmware.com    2.9.1+vmware.1-tkg.1    Reconcile succeeded   6d22h
   pkg_namespace=mypackage
   
   # Update the package
   tanzu -n ${pkg_namespace} package installed update harbor --ytt-overlay-file pkg-harbor-overlay.yaml
   #> ...
   #> hh:mm:ss: Deploy succeeded

4. Verify the result
   
   

   ## Check - all Harbor pods are at "Running" state
   kubectl -n tanzu-system-registry get all
   
   ## Check - Deployment
   kubectl -n tanzu-system-registry get deploy harbor-jobservice -oyaml | grep -A1 'name: REGISTRY_HTTP_CLIENT_TIMEOUT'
       - name: REGISTRY_HTTP_CLIENT_TIMEOUT
         value: "60"
   
   ## Check - Pod
   pod=$(kubectl -n tanzu-system-registry get pod -l app=harbor,component=jobservice -o jsonpath='{.items[0].metadata.name}')
   kubectl -n tanzu-system-registry exec $pod -- env | grep REGISTRY_HTTP_CLIENT_TIMEOUT
   REGISTRY_HTTP_CLIENT_TIMEOUT=60

5.  Revert the Harbor registry READ ONLY mode
   
   Harbor WebUI --> Administration --> Configuration --> System Settings --> Uncheck "Repository Read Only" --> SAVE

Additional Information

You can check the current YTT overlay YAML file contents using below command:

kubectl -n ${pkg_namespace} get secrets harbor-${pkg_namespace}-overlays -oyaml | yq '.data["0000-pkg-harbor-overlay.yaml"]' | base64 -d