Cert-Manager Package Reconciliation Fails with ImagePull Error and “Unknown Blob” Message in Tanzu Kubernetes Grid Service with Harbor Registry

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

This article explains an issue where the cert-manager package reconciliation fails in Tanzu Kubernetes Grid Service (vSphere with Tanzu) environments that use a Harbor registry as the package repository.

Administrators may observe that the cert-manager deployment fails due to an ImagePull error and unknown blob messages when fetching package images from Harbor.

This issue typically occurs when the Harbor repository reference used by the Tanzu package repository is invalid or no longer exists, even though the image name still appears in the Harbor UI.

• The cert-manager package reconciliation fails.
• The package shows as reconciling or failed when listing packages using tanzu package installed list -A.
• ImagePull failures with unknown blob errors appear in logs.
• The impacted image appears in Harbor UI but cannot be viewed or pulled.
• Manually pulling the image using crictl pull or docker pull

Observed Behavior: The kubectl get apps -A command shows the following for affected packages:

 

NAMESPACE           NAME                 DESCRIPTION                                                                       

cert-manager-ns     cert-manager         Reconcile failed: Deploying: Error (see .status.usefulErrorMessage for details)   

 

The kubectl get pod -A | grep cert output indicates:

 

NAMESPACE           NAME                                    STATUS             

cert-manager-ns     cert-manager-<hash>                    0/1 ImagePullBackOff 

cert-manager-ns     cert-manager-cainjector-<hash>         0/1 ImagePullBackOff 

cert-manager-ns     cert-manager-webhook-<hash>            0/1 ImagePullBackOff 

 

from kubectl describe apps:

 

fail: reconcile deployment/cert-manager (apps/v1) namespace: cert-manager-ns

^ Deployment is not progressing: ProgressDeadlineExceeded, message: ReplicaSet "cert-manager-<hash>" has timed out progressing.

Fetch:

  Exit Code:   0

  Stdout:      apiVersion: vendir.k14s.io/v1alpha1

  directories:

  - contents:

    - imgpkgBundle:

        image: <internal-harbor-registry>/vks3.3.1/tanzu-packages/packages/standard/repo@sha256:<hash>

      path: .

    path: "0"

  kind: LockConfig

Friendly Description:  Reconcile failed: Deploying: Error (see .status.usefulErrorMessage for details)

Useful Error Message:  kapp: Error: waiting on reconcile deployment/cert-manager (apps/v1) namespace: cert-manager-ns:

  Finished waiting unsuccessfully:

    Deployment is not progressing:

      ProgressDeadlineExceeded, message:

        ReplicaSet "cert-manager-<hash>" has timed out progressing.

Events:  <none>

Pod Describe Output:

QoS Class:       BestEffort

Node-Selectors:  kubernetes.io/os=linux

Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

  Type     Reason   Age                   From     Message

  ----     ------   ----                  ----     -------

  Normal   BackOff  116s (x14851 over 2d8h)  kubelet  Back-off pulling image "<internal-harbor-registry>/vks3.3.1/tanzu-packages/packages/standard/repo@sha256:<hash>"

  Warning  Failed   116s (x14851 over 2d8h)  kubelet  Error: ImagePullBackOff

Manual Image Pull Attempt: A manual attempt to pull the image from a bootstrap machine resulted in:

$ docker pull <internal-harbor-registry>/vks3.3.1/tanzu-packages/packages/standard/repo@sha256:<hash>

<internal-harbor-registry>/vks3.3.1/tanzu-packages/packages/standard/repo@sha256:<hash>: Pulling from vks3.3.1/tanzu-packages/packages/standard/repo

<layer-hash>: Pull complete

<layer-hash>: Pull complete

...

unknown blob

 

Environment

• Tanzu Kubernetes Grid Service
• VMware Kubernetes Service (VKS) 
• Harbor Registry (embedded or external)

Cause

This issue occurs when the Tanzu package repository is pointing to a Harbor repository path that no longer exists or is misconfigured.

Although the image may still appear under the Harbor project, its associated content blobs are missing or inaccessible. As a result, image pulls fail with an “unknown blob” error.

Typical scenarios that can cause this include:

• Repository structure changed during an upgrade or migration.
• Old repository references not updated in the package configuration.
• Garbage collection (GC) performed in Harbor that removed unreferenced blobs.

Resolution

To resolve this issue, update the Tanzu package repository to point to a valid Harbor repository and reinstall the cert-manager package.

Step 1 – List existing repositories

tanzu package repository list -A

Step 2 – Update the repository URL

Update the repository to the correct Harbor path that contains valid images.

tanzu package repository update standard-repo \--url ####.####.####. /####/####/#####/ \-n #####

Step 3 – Remove the existing cert-manager package

tanzu package installed delete cert-manager --namespace #####

Step 4 – Reinstall the cert-manager package

tanzu package install cert-manager \-p cert-manager.kubernetes.vmware.com \--namespace ##### \--version #.##.#+vmware.1-vks.1

Step 5 – Verify successful reconciliation

Run the following command to ensure the package installation is successful:

tanzu package installed list -A

The status should show “Reconcile succeeded” for the cert-manager package.