Fluent-bit Package Reconciliation Fails with ImagePull Error and “Unknown Blob” Message in Tanzu Kubernetes Grid Service with Harbor Registry

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

^{This article addresses an issue where the Fluent Bit package reconciliation remains in a pending state in a Tanzu Kubernetes Grid (TKG) environment. The root cause was traced to a missing image layer in the Harbor container registry, resulting in image pull failures during package deployment.}

^Issue

^{The Fluent Bit package was stuck in a pending state and failed to reconcile successfully.}

^{Observed Behavior}

• ^{Fluent Bit pods failed to start.}
• ^{Image pull attempts returned errors referencing a missing image layer.}
• ^{Manual image pulls using crictl pull also failed with BLOB_UNKNOWN and descriptor not found errors.}

^{Attempts to pull the image manually using container runtimes such as crictl pull resulted in the following error:}

^{failed to copy: httpReadSeeker: failed open: could not fetch descriptor sha256:###### (application/vnd.docker.image.rootfs.diff.tar.gzip) from remote: not found}

Environment

• ^{Tanzu Kubernetes Grid Service}
• ^{VMware Kubernetes Service (VKS)} 
• ^{Harbor Registry (embedded or external)}

Cause

^{A critical image layer required for the Fluent Bit package was missing from the Harbor registry. Although the image manifest and some layers were present.}

Resolution

^{Recommended Remediation Steps:}

 

        ^{1. Pull a Fresh Copy of the Image}

            ^{Retrieve the correct image from the public registry.}

        ^{2. Remove the Corrupted Image from Harbor}

            ^{Delete the incomplete or corrupted image from the Harbor registry to prevent further pull attempts from failing.}

        ^{3. Push the Fresh Image to Harbor}

            ^{Re-upload the complete image, ensuring all layers are present.}

        ^{4. Update the Package Repository Reference}

            ^{Confirm that the Fluent Bit package references the correct and complete image.}

        ^{5. Retry the Fluent Bit Package Installation}

            ^{Re-initiate the package deployment to verify successful reconciliation.}

Additional Information

^{For more information on Fluent Bit package deployment, refer to the official Broadcom documentation: 🔗 https://techdocs.broadcom.com/us/en/vmware-tanzu/standalone-components/tanzu-kubernetes-grid/2-5/tkg/workload-packages-packages-fluentbit-mc.html}