Error "ReconcileFailed. MANIFEST_UNKNOWN: manifest unknown" in Velero vSphere Operator service

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The Supervisor Cluster upgrade process stalls.
• The update successfully deploys new supervisor VMs and installs agents on all ESXi hosts, but it is stuck unable to fetch the Velero Image.
• The following error appears in the svc-velero.vsphere.vmware.com PackageInstall:
      vendir: Error: Syncing directory '0':
      Syncing directory '.' with imgpkgBundle contents:
        Fetching image: 
        GET https://docker-registry.kube-system.svc:5000/v2/velero-vsphere/package/velero-vsphere/manifests/1.6.1-23741747:
            MANIFEST_UNKNOWN: manifest unknown; map[Tag:1.6.1-23741747]
  version: 1.6.1-embedded+23741747

• Error as seen from the vSphere UI:

• When you view the Velero deployment or pod manifest on the Supervisor cluster, you see the image it attempts to pull.

> docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>

• When you log in to any Supervisor control plane node and attempt a manual pull, you receive a "not found" error.

> crictl --debug pull docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>

E0521 19:13:17.455815 3535847 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>\": failed to resolve reference \"docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>\": docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>: not found" image="docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>"
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>": failed to resolve reference "docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>": docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>: not found

Environment

• vCenter 8.0.x
• VMware vSphere Kubernetes Service

Cause

• An invalid Velero vSphere Operator service version is being used.
• The embedded version does not have a valid manifest.

Resolution

1. Log in to the vSphere UI.

2. Click Menu > Workload Management > Services.

3. Select the Velero vSphere Operator service.

4. Click Manage.

5. Click the Install Version drop-down menu.

6. Note the installed version.

7. Choose the most recent embedded version to use.

8. Allow the service to reconcile.
   
   Example dropdown: