Manual workload cluster upgrade fails with KR/OSImage resolution error

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When attempting a manual Kubernetes version upgrade for a workload cluster using the CLI, the tkr-resolver-cluster-webhook admission webhook may deny the request. This typically occurs during kubectl edit cluster operations when the Tanzu Kubernetes Release (TKR) resolver cannot uniquely identify a compatible Virtual Machine Image (VMI) in the vCenter Content Library. Users may encounter "Multiple OSImages Resolved" or "Missing compatible KR/OSImage" errors during this process.

Cause

These errors are caused by how the Tanzu Kubernetes Release (TKR) resolver handles node image selection during a cluster update:

• Multiple OSImages Conflict: The vCenter Content Library contains multiple matching Virtual Machine Images (VMIs) for the exact same target Kubernetes version. For example, if both Ubuntu 22.04 and Ubuntu 24.04 templates exist for K8s v1.33.6, the default filter (os-name=ubuntu) matches both. The admission control blocks the upgrade to avoid guessing the incorrect OS architecture.

• Missing Compatible OSImage: This occurs in two common scenarios. First, an os-version annotation is added to the cluster manifest, but the spec.topology.version is not updated to the new Kubernetes release in the exact same patch. Second, the selected OS version exists in the content library but is not marked as fully compatible by the Supervisor for that specific Kubernetes release.

Resolution

To resolve these conflicts, manually patch the cluster manifest to explicitly define the target Kubernetes version and distinguish the exact OS version using the resolve-os-image annotation in a single transaction.

1. Connect to the vSphere Supervisor namespace using kubectl.

2. Verify available images and compatibility for the target Kubernetes version by running the following command:
   kubectl get cclitem -A | grep v1.33.6

3. Identify an OS version marked as fully compatible by locating True true in the final columns of the command output (e.g., True false vs. True true).

4. Edit the target cluster manifest by running the following command:
   kubectl edit cluster <cluster-name> -n <namespace>

5. Update the version: value to the target Kubernetes version under the spec.topology section.

6. Add the specific OS image resolver annotation (e.g., run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu,os-version=24.04) to the metadata.annotations section under controlPlane.

7. Add the exact same annotation to the metadata.annotations section under workers.machineDeployments.

8. Save the file and exit the text editor.

Example YAML Configuration:

YAML

 

spec:
  topology:
    class: builtin-generic-v3.6.0
    # Step 5: Update target version here
    version: v1.33.6---vmware.1-fips-vkr.2
    controlPlane:
      metadata:
        annotations:
          # Step 6: Add specific OS image resolver annotation here
          run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu,os-version=24.04
      replicas: 3
    workers:
      machineDeployments:
        - class: node-pool
          name: worker-pool-1
          metadata:
            annotations:
              # Step 7: Add the exact same annotation to the worker nodes
              run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu,os-version=24.04

Explicitly declaring the OS version alongside the updated Kubernetes version in a single API patch allows the admission webhook to successfully filter out the conflicting template and initiate the rolling upgrade.