Guest Cluster Upgrade Fails with TKR Resolver Error Due to OS Image Annotations

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Upgrading or modifying a Tanzu Kubernetes guest cluster that uses the ClusterClass (v1beta1) API may fail with an admission webhook error from the Tanzu Kubernetes Release (TKR) resolver.
  
  
• During an upgrade attempt (for example, when changing the Kubernetes version), the update to the Cluster resource is rejected before any rollout occurs with the below error:

error: clusters.cluster.x-k8s.io <guest-cluster-name> could not be patched:
admission webhook "tkr-resolver-cluster-webhook.tanzu.vmware.com" denied the request:
Could not resolve KR/OSImage. Missing compatible KR/OSImage for the cluster
Control Plane, filters: {k8sVersionPrefix: vX.Y.Z+vmware.A-..., osImageSelector: name=vmi-######}
MachineDeployment <machine-deployment-name>, filters: {k8sVersionPrefix: vX.Y.Z+vmware.A-..., osImageSelector: name=vmi-######}

• • The update operation on the Cluster resource is rejected by the TKR resolver admission webhook.

  • The error references:

    • k8sVersionPrefix – the requested Kubernetes version.

    • osImageSelector: name=vmi-###### – a specific VM image identifier.

  • As the admission request is denied, no control plane or worker nodes begin upgrading.

Environment

• VMware vSphere Kubernetes Service

Cause

• The issue occurs when the Cluster specification contains OS image annotations that explicitly reference a specific VM Image (VMI) ID.

Example configuration:

spec:
  topology:
    classRef:
      name: builtin-generic
      namespace: vmware-system-vks-public
    version: vX.Y.Z+vmware.A-...
    controlPlane:
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: name=vmi-0fbbe6ac93e81e1d4
    workers:
      machineDeployments:
      - class: node-pool
        metadata:
          annotations:
            run.tanzu.vmware.com/resolve-os-image: name=vmi-0fbbe6ac93e81e1d4

• In this configuration:

  • The annotation run.tanzu.vmware.com/resolve-os-image specifies name=<vmi-id>.

  • This pins the cluster to a specific VM image.

• During an upgrade, the TKR resolver must find a compatible combination of:

  • A Kubernetes Release (TKR) matching the requested version, and

  • An OSImage matching the specified selector.

• If the requested Kubernetes version does not have a mapping for that exact VM image, the resolver cannot determine a valid TKR/OSImage combination and rejects the request.
• As a result:

• • The Kubernetes version requested may be valid.

  • However, it cannot be used with the pinned VM image, causing the upgrade to fail.

Resolution

Remove the VM image–specific selector and instead configure the cluster to use an OS family selector. This allows the resolver to automatically choose a compatible OS image for the requested Kubernetes version.

1. Update the Cluster OS Image Annotation
   • Edit the Cluster resource and replace the VMI-specific selector with an OS family selector.

Example configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <guest-cluster-name>
  namespace: <guest-cluster-namespace>
spec:
  topology:
    classRef:
      name: builtin-generic
      namespace: vmware-system-vks-public
    version: vX.Y.Z+vmware.A-...
    controlPlane:
      replicas: 3
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu       <------- Removed the VM image selector
    workers:
      machineDeployments:
      - class: node-pool
        name: md-0
        metadata:
          annotations:
            run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu

1. • This allows the resolver to select any compatible OSImage for the requested Kubernetes version.

2. Apply the Updated Configuration
   • If the cluster was edited using kubectl edit, save the updated configuration after modifying the annotation.
   • If using the temporary file referenced in the error message, apply the updated manifest:

kubectl replace -f /tmp/kubectl-edit-####.yaml

• • Once the update is accepted, the controllers will reconcile the cluster and begin the upgrade process.

Additional Information

Best Practices

• Avoid pinning clusters to a specific VM image ID (VMI) unless strictly required.

• Use OS family selectors such as os-name=ubuntu or os-name=photon to maintain upgrade flexibility.

• Before performing cluster upgrades, ensure that compatible TKRs and OSImages are available for the target Kubernetes version.

• In environments with strict image management policies, verify that new OS images are published and mapped to TKRs before initiating upgrades.