"ReconcileFailed. MANIFEST_UNKNOWN: manifest unknown" error shown in Velero vSphere Operator service

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Scenario:

• During the update operation, Supervisor Cluster Update has stalled.
  
  
• The update successfully deploys new supervisor VMs and install agents on all ESXi hosts. But has been stuck here, unable to fetch Velero Image.
  
  
• The error as seen from the svc-velero.vsphere.vmware.com PackageInstall:

    vendir: Error: Syncing directory '0':
      Syncing directory '.' with imgpkgBundle contents:
        Fetching image:
          GET https://docker-registry.kube-system.svc:5000/v2/velero-vsphere/package/velero-vsphere/manifests/1.6.1-23741747:
            MANIFEST_UNKNOWN: manifest unknown; map[Tag:1.6.1-23741747]
  version: 1.6.1-embedded+23741747

 

• Error as seen from the vSphere UI:

 

• When viewing either Velero deployment or Velero pod manifest on the Supervisor cluster, you note the Image that is attempted to be pulled:

For example:

docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>

And when you log in to any Supervisor control plane node, and attempt a manual pull you receive "not found" error.

Example:

crictl --debug pull docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>
    
E0521 19:13:17.455815 3535847 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>\": failed to resolve reference \"docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>\": docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>: not found" image="docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>"
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>": failed to resolve reference "docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>": docker-registry.kube-system.svc:5000/velero-vsphere/package/velero-vsphere@sha256:<IMAGE_SHA_STRING>: not found

Environment

vCenter 8.0.3 Ue

VMware vSphere Kubernetes Service

Cause

An invalid Velero vSphere Operator service version is being used.  The embedded version used does not have a valid manifest.

Resolution

• Login to vSphere UI:
  
  
• Select the Velero vSphere Operator service
  
  
• Select Manage
  
  
• Select the Install Version dropdown
  
  
• Note the installed version
  
  
• Choose the most recent embedded version to use
  
  
• Allow it reconcile back

Example dropdown: