v1beta1 Cluster Upgrade to v1.28.8 Fails Due to Unsupported OS Version in Annotation

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Attempting to upgrade a v1beta1 cluster from v1.27.11+vmware.1-fips.1-tkg.2 to v1.28.8+vmware.1-fips.1-tkg.2 using kubectl edit cluster fails. The error returned from the tkr-resolver-cluster-webhook indicates it cannot resolve a compatible OSImage for the control plane and worker nodes due to version mismatch.

kubectl edit cluster <cluster> -n <namespace>

error: clusters.cluster.x-k8s.io "<cluster>" could not be patched: admission webhook "tkr-resolver-cluster-webhook.tanzu.vmware.com" denied the request: could not resolve KR/OSImage for controlPlane, machineDeployments: [workers], query: {controlPlane: {k8sVersionPrefix: "v1.28.8+vmware.1-fips.1-tkg.2", krSelector: "tkr.tanzu.vmware.com/standard", osImageSelector: "os-arch=amd64,os-name=photon,os-version=3,tkr.tanzu.vmware.com/standard"}, machineDeployments: [{k8sVersionPrefix: "v1.28.8+vmware.1-fips.1-tkg.2", krSelector: "tkr.tanzu.vmware.com/standard", osImageSelector: "os-arch=amd64,os-name=photon,os-version=3"}]}, result: {controlPlane: {k8sVersion: "", krName: "", osImagesByKR: map[]}, machineDeployments: [{k8sVersion: "", krName: "", osImagesByKR: map[]}]}

Environment

• TKG Service 3x
• Current Kubernetes release version: v1.27.11+vmware.1-fips.1-tkg.2
• Target Kubernetes release version: v1.28.8+vmware.1-fips.1-tkg.2
• Supervisor 8
• VMware vSphere with Tanzu 

Cause

The Cluster object includes an annotation specifying os-version=3. However, the TKR for v1.28.8+vmware.1-fips.1-tkg.2 only includes an OSImage for Photon OS version 5. As a result, the upgrade fails because the resolver cannot find a matching OSImage.

Resolution

1. Edit the Cluster object and remove the os-version=3 reference from the annotation.
   • Change:
     run.tanzu.vmware.com/resolve-os-image: os-name=photon,os-version=3,os-arch=amd64
     To:
     run.tanzu.vmware.com/resolve-os-image: os-name=photon,os-arch=amd64
2. Save the Cluster object.
3. After completing these steps, retry the upgrade operation.