Supervisor cluster upgrade failed at Component ImageRegistry with 'ImageRegistryUpgrade failed'

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Upgrade of Supervisor Cluster from v1.27.5+vmware.1-fips.1-vsc0.1.8-23388239 to v1.28.3+vmware.2-fips.1-vsc0.1.10-24224934.
The update failed on ImageRegistryUpgrade failed.

ImageRegistry configured
Configuration error (since #/##/2025, #:##:## AM)
Component Configuration error: Component ImageRegistryUpgrade failed: Failed to run command: ['kubectl', 'apply', '-f', '/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml', '--record'] ret=1 out= err=Flag --record has been deprecated, --record will be removed in the future warning: error calculating patch from openapi spec: expected kind, but got map Error from server (ServiceUnavailable): error when applying patch: {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"data.packaging.carvel.dev/v1alpha1\",\"kind\":\"Package\",\"metadata\":{\"annotations\":{\"kubernetes.io/change-cause\":\"kubectl apply --filename=/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml --record=true\"},\"name\":\"vmimages.vmware.com.1.0.0+vmware.1\",\"namespace\":\"vmware-system-imageregistry\"},\"spec\":{\"refName\":\"vmimages.vmware.com\",\"releaseNotes\":\"VM image operator package\\n\",\"template\":{\"spec\":{\"deploy\":[{\"kapp\":{}}],\"fetch\":[{\"imgpkgBundle\":{\"image\":\"localhost:5000/vmware/imageregistry-operator/package@sha256:################################################################\"}}],\"template\":[{\"ytt\":{\"paths\":[\"config/\"]}},{\"kbld\":{\"paths\":[\".imgpkg/images.yml\",\"-\"]}}]}},\"valuesSchema\":{\"openAPIv3\":{\"additionalProperties\":false,\"properties\":{\"fss_importovf_enabled\":{\"default\":\"false\",\"type\":\"string\"},\"fss_multiple_cl_enabled\":{\"default\":\"false\",\"type\":\"string\"},\"namespace\":{\"default\":\"vmware-system-imageregistry\",\"type\":\"string\"},\"replicas\":{\"default\":2,\"type\":\"integer\"}},\"type\":\"object\"}},\"version\":\"1.0.0+vmware.1\"}}\n"}},"spec":{"template":{"spec":{"fetch":[{"imgpkgBundle":{"image":"localhost:5000/vmware/imageregistry-operator/package@sha256:################################################################"}}]}},"valuesSchema":{"openAPIv3":{"properties":{"fss_multiple_cl_enabled":{"default":"false","type":"string"}}}}}} to: Resource: "data.packaging.carvel.dev/v1alpha1, Resource=packages", GroupVersionKind: "data.packaging.carvel.dev/v1alpha1, Kind=Package" Name: "vmimages.vmware.com.1.0.0+vmware.1", Namespace: "vmware-system-imageregistry" for: "/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml": error when patching "/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml": the server is currently unable to handle the request (patch packages.data.packaging.carvel.dev vmimages.vmware.com.1.0.0+vmware.1) Error from server (ServiceUnavailable): error when retrieving current configuration of: Resource: "data.packaging.carvel.dev/v1alpha1, Resource=packagemetadatas", GroupVersionKind: "data.packaging.carvel.dev/v1alpha1, Kind=PackageMetadata" Name: "vmimages.vmware.com", Namespace: "vmware-system-imageregistry" from server for: "/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml": the server is currently unable to handle the request (get packagemetadatas.data.packaging.carvel.dev vmimages.vmware.com)

/usr/lib/vmware-wcp/upgrade/upgrade-ctl.py get-status | jq '.progress | to_entries | .[] | "\(.value.status) - \(.key)"' | sort
"failed - ImageRegistryUpgrade"
"skipped - CertManagerAdditionalUpgrade"
"skipped - EmbeddedTKGServiceCleanup"
"skipped - LoadBalancerApiUpgrade"
"skipped - VmOperatorUpgrade"
"upgraded - AKOUpgrade"
"upgraded - AppPlatformOperatorUpgrade"
"upgraded - CertManagerUpgrade"
....

error: Component ImageRegistryUpgrade failed was observed.

Failed to run command: ['kubectl', 'apply', '-f', '/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml', '--record'] ret=1 out= err=Flag --record has been deprecated, --record will be removed in the future warning: error calculating patch from openapi spec: expected kind, but got map Error from server (ServiceUnavailable): error when applying patch

Environment

VKS
Supervisor K8s from 1.27.5 to 1.28.3

Cause

The error appears to advise that the package yaml was not available locally at time of upgrade.
Also appears that the image fetch could not be handled on the server side as per this error:

Package image fetch below not available at time of upgrade
[{\"imgpkgBundle\":{\"image\":\"localhost:5000/vmware/imageregistry-operator/package@sha256:##############################################\"}}]

Server side errored out when image was being fetched

{"fetch":[{"imgpkgBundle":{"image":"localhost:5000/vmware/imageregistry-operator/package@sha256:##############################################"}}]}},"valuesSchema":{"openAPIv3":{"properties":{"fss_multiple_cl_enabled":{"default":"false","type":"string"}}}}}} to: Resource: "data.packaging.carvel.dev/v1alpha1, Resource=packages", GroupVersionKind: "data.packaging.carvel.dev/v1alpha1, Kind=Package" Name: "vmimages.vmware.com.1.0.0+vmware.1", Namespace: "vmware-system-imageregistry" for: "/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml": error when patching "/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml": the server is currently unable to handle the request (patch packages.data.packaging.carvel.dev vmimages.vmware.com.1.0.0+vmware.1) Error from server (ServiceUnavailable): error when retrieving current configuration of: Resource: "data.packaging.carvel.dev/v1alpha1, Resource=packagemetadatas", GroupVersionKind: "data.packaging.carvel.dev/v1alpha1, Kind=PackageMetadata" Name: "vmimages.vmware.com", Namespace: "vmware-system-imageregistry" from server for: "/usr/lib/vmware-wcp/objects/image-registry-operator/package/package.yaml": the server is currently unable to handle the request (get packagemetadatas.data.packaging.carvel.dev vmimages.vmware.com)

Resolution

Reattempt upgrade for a second time and it will complete on second attempt.
Previous cases confirmed that triggering the upgrade for a 2nd time helped solve.