VKS Workload Cluster Upgrade Failure from v1.34.2+vmware.2 to v1.35.2+vmware.1

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When upgrading a VKS (vSphere Kubernetes Service) workload cluster from v1.34.1 to v1.35.2 fails
• No new Kubernetes nodes are created after updating the cluster's version to v1.35.2
• Describing the cluster shows the following error:

    Last Transition Time:  <TIMESTAMP>
    Message:               * TopologyReconciled: error computing the desired state of the Cluster topology: failed to compute ControlPlane: failed to compute version of ControlPlane: failed to call extension handlers for hook "BeforeClusterUpgrade.hooks.runtime.cluster.x-k8s.io": failed to call extension handler "before-cluster-upgrade.runtime-extension": got failure response, please check controller logs for errors
    Observed Generation:   8
    Reason:                NotAvailable
    Status:                False
    Type:                  Available
    Last Transition Time:  <TIMESTAMP>

• The CAPI manager controller logs have:
  

  <TIMESTAMP> stderr 1 client.go:303] "failed to call extension handlers" err="failed to call extension handler \"before-cluster-upgrade.runtime-extension\": got failure response, please check controller logs for errors" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="<NAMESPACE>/<CLUSTER_NAME>" reconcileID="<UUID>" hook="BeforeClusterUpgrade"

  <TIMESTAMP> stderr 1 controller.go:474] "Reconciler error" err="error computing the desired state of the Cluster topology: failed to compute ControlPlane: failed to compute version of ControlPlane: failed to call extension handlers for hook \"BeforeClusterUpgrade.hooks.runtime.cluster.x-k8s.io\": failed to call extension handler \"before-cluster-upgrade.runtime-extension\": got failure response, please check controller logs for errors" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="<NAMESPACE>/<CLUSTER_NAME>" reconcileID="<UUID>"

Environment

VKS 3.6.2+v1.35 and earlier. 

This issue has been resolved in VKS 3.6.3 and later.

Cause

An AddonRelease has a constraint preventing the upgrade.

Resolution

Update VKS to v3.6.3 or later. If you're unable to update VKS, you can apply the fix to all clusters in a VKS namespace or to specific clusters. 

Apply the fix to all clusters 

1. Log into the supervisor as an administrator (e.g., [e-mail])
2. Create a file named addoninstall.yaml with the following contents. <supervisor-namespace> is the namespace that contains the clusters you want to apply the fix to:
   

   apiVersion: addons.kubernetes.vmware.com/v1alpha1
   kind: AddonInstall
   metadata:
     labels:
       addon.kubernetes.vmware.com/addon-name: antrea
       addons.kubernetes.vmware.com/category: cni
     name: addoninstall-antrea-v1.35.2---vmware.1-vkr.3
     namespace: <supervisor-namespace>
   spec:
     addonRef:
       name: antrea
     clusters:
     - constraints:
         expression: cluster.cniRefName() == 'antrea'
       selector:
         matchLabels:
           run.tanzu.vmware.com/tkr: v1.35.2---vmware.1-vkr.3
     crossNamespaceSelection: Blocked
     releaseFilter:
       ref:
         name: antrea.tanzu.vmware.com-2.5.1-vmware.2-tkg.1---v1.35.2-vmware.1-vkr.3
         namespace: vmware-system-vks-public
     stopMatchingBehavior: Retain

3. Create the AddonInstall
   

   kubectl apply -f addoninstall.yaml

4. Within a few minutes you should see new Kubernetes nodes being created

Apply the fix to a specific cluster 

1. Log into the supervisor as an administrator (e.g., [e-mail])
2. Create a file named addoninstall.yaml with the following contents. <supervisor-namespace> is the namespace that contains the cluster you want to apply the fix to. <workload-cluster> is the name of the cluster:
   

   apiVersion: addons.kubernetes.vmware.com/v1alpha1
   kind: AddonInstall
   metadata:
     labels:
       addon.kubernetes.vmware.com/addon-name: antrea
       addons.kubernetes.vmware.com/category: cni
     name: addoninstall-antrea-v1.35.2---vmware.1-vkr.3
     namespace: <supervisor-namespace>
   spec:
     addonRef:
       name: antrea
     clusters:
     - constraints:
         expression: cluster.cniRefName() == 'antrea'
       selector:
         matchLabels:
           cluster.x-k8s.io/cluster-name: <workload-cluster>
     crossNamespaceSelection: Blocked
     releaseFilter:
       ref:
         name: antrea.tanzu.vmware.com-2.5.1-vmware.2-tkg.1---v1.35.2-vmware.1-vkr.3
         namespace: vmware-system-vks-public
     stopMatchingBehavior: Retain

3. Create the AddonInstall
   

   kubectl apply -f addoninstall.yaml

4. Within a few minutes you should see new Kubernetes nodes being created