After upgrading TKC/VKS cluster, no new VMs are created

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After applying the Yaml file with newer version, there are no new VMs created. 

Environment

VKS 3.3.0

Cause

The Target VM image are unable to download. 

 

TKG-Controller Log

stderr F I0527 07:08:43.666060       1 tanzukubernetescluster_controller.go:621] "TanzuKubernetesCluster is not yet ready to retire" logger="#####" cluster="#####" reason="retirement is waiting for Kubernetes Release of TKC and Cluster to match"
stderr F E0527 07:08:44.054795       1 tanzukubernetescluster_controller.go:807] "Error while reconciling cluster object requeuing for retry" err=<
stderr F         admission webhook "tkr-resolver-cluster-webhook.tanzu.vmware.com" denied the request: Could not resolve KR/OSImage
tderr F  Missing compatible KR/OSImage for the cluster
stderr F         Control Plane, filters: {k8sVersionPrefix: v1.31.14+vmware.1-fips-vkr.2, osImageSelector: os-name=photon,tkr.tanzu.vmware.com/standard}
stderr F         MachineDeployment worker-nodepool, filters: {k8sVersionPrefix: v1.31.14+vmware.1-fips-vkr.2, osImageSelector: os-name=photon}
stderr F  > logger="#####" cluster.name="#####"

Check the content library item, the ova file are not downloaded

kubectl get contentlibraryitems
NAME                       VSPHERENAME                                                                 CLUSTERCONTENTLIBRARYREF   TYPE   READY   CACHED   SIZEINBYTES   SECURITYCOMPLIANT   AGE
clitem-1d43760d24a18d29d   ob-25155952-photon-5-amd64-v1.31.14---vmware.1-fips-vkr.2                   cl-c92ab8bce673f8f3d       OVF    False   false                  true                76d
clitem-478dd7ae37aba83c2   ob-25155953-ubuntu-2204-amd64-v1.31.14---vmware.1-fips-vkr.2                cl-c92ab8bce673f8f3d       OVF    True    true     8670776215    true                76d

kubectl get clustervirtualmachineimages
NAME                    DISPLAY NAME                                                                IMAGE VERSION                           OS NAME               OS VERSION   HARDWARE VERSION
vmi-1d43760d24a18d29d   ob-25155952-photon-5-amd64-v1.31.14---vmware.1-fips-vkr.2
vmi-478dd7ae37aba83c2   ob-25155953-ubuntu-2204-amd64-v1.31.14---vmware.1-fips-vkr.2                v1.31.14+vmware.1-fips-vkr.2            ubuntu64Guest                      17

Resolution

Need to download the OVA file manually.