vSphere Supervisor ImageRegistry-Operator Reconcile Failed state due to MANIFEST_UNKNOWN

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In the Supervisor cluster, the system pod imageregistry-operator may not be affected by this error if the existing imageregistry-operator pod is still running.

However, when viewing and describing the corresponding PKGi, it is found to be in ReconcileFailed state with the below error message:

kubectl get pkgi -A
vmware-system-imageregistry         packageinstall.packaging.carvel.dev/imageregistry-operator                         vmimages.vmware.com                                  #.#.#+vmware.#                 Reconcile failed: Error (see .status.usefulErrorMessage for details)

kubectl describe pkgi -n vmware-system-imageregistry imageregistry-operator
Fetching image:
      GET 
https://localhost:5000/v2/vmware/imageregistry-operator/package/manifests/sha256:<omitted>: 
       MANIFEST_UNKNOWN: manifest unknown; 
map[Name:vmware/imageregistry-operator/package 
Revision:sha256:<omitted>]

 

Viewing the logs for the docker-registry system pod shows errors similar to the following referencing the above missing manifest:

kubectl logs -n kube-system <docker-registry-pod>

level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=vmware/imageregistry-operator/package revision=sha256:<omitted>" err.message="manifest unknown" go.version="go1.22.5 X:boringcrypto" http.request.host="127.0.0.1:5002" http.request.id=<omitted> http.request.method=GET http.request.remoteaddr="<localhost>:<port>" http.request.uri="/v2/vmware/imageregistry-operator/package/manifests/sha256:<omitted>" http.request.useragent="go-containerregistry/v0.16.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.074192ms http.response.status=404 http.response.written=213 vars.name="vmware/imageregistry-operator/package" vars.reference="sha256:<omitted>"

 

Environment

vSphere Supervisor

Cause

The manifest has gone missing from the Supervisor cluster.

Resolution

Reach out to VMware by Broadcom Technical Support to validate and perform steps to resolve this issue.

Additional Information

• • Japanese version: vSphere Supervisor の ImageRegistry-Operator が MANIFEST_UNKNOWN エラーにより Reconcile Failed 状態になっている問題