VKS Cluster Upgrade Failure to v1.35.X Stuck TopologyReconciled Failed/False

Products

VMware vSphere Kubernetes Service

Issue/Introduction

Upgrading a VKS (vSphere Kubernetes Service) cluster from v1.34.x to v1.35.x fails with the following symptoms:

While connected to the Supervisor cluster context, the following symptoms are observed:

No new nodes or VMs are created after updating the cluster's version to v1.35.x:
```
kubectl get machines,vm -n <affected vks cluster namespace>
```
The cluster and clusterbootstrap are on the desired version of v1.35.x, but the kubeadmcontrolplane (kcp) and machineDeployment (md) are still on the old version of v1.34.x:
```
kubectl get cluster,clusterbootstrap,kcp,md -n <affected vks cluster namespace>
```

Describing the affected VKS cluster shows the following error:

kubectl describe cluster -n <affected vks cluster namespace> <vks cluster name>

    Message:               * TopologyReconciled: error computing the desired state of the Cluster topology: failed to compute ControlPlane: failed to compute version of ControlPlane: failed to call extension handlers for hook "BeforeClusterUpgrade.hooks.runtime.cluster.x-k8s.io": failed to call extension handler "before-cluster-upgrade.runtime-extension": got failure response, please check controller logs for errors
    Observed Generation:   8
    Reason:                NotAvailable
    Status:                False
    Type:                  Available

CAPI manager controller logs show the following errors:

"failed to call extension handlers" err="failed to call extension handler \"before-cluster-upgrade.runtime-extension\": got failure response, please check controller logs for errors" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="<NAMESPACE>/<VKS_CLUSTER_NAME>" reconcileID="<UUID>" hook="BeforeClusterUpgrade"

"Reconciler error" err="error computing the desired state of the Cluster topology: failed to compute ControlPlane: failed to compute version of ControlPlane: failed to call extension handlers for hook \"BeforeClusterUpgrade.hooks.runtime.cluster.x-k8s.io\": failed to call extension handler \"before-cluster-upgrade.runtime-extension\": got failure response, please check controller logs for errors" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="<NAMESPACE>/<VKS_CLUSTER_NAME>" reconcileID="<UUID>"

tanzu-addons-controller logs show the below error:

"failed to selectAddonRelease" err="expected to find at least one version, but did not (details: all=4 -> after-kuberntes-version-check=0) -> after-upgrade-version-check=0 failed to select item" logger="controllers.ClusterAddon" clusterAddon.Name="<VKS cluster name>-antrea" clusterAddon.Namespace="<VKS cluster namespace>"

The above noted ClusterAddon is in Unknown state:

kubectl get clusteraddon -n <vks cluster namespace>

However, the addonRelease of antrea for the desired v1.35.x VKR version exists in the affected cluster's namespace:
```
kubectl get addonreleases -n <vks cluster namespace> | grep antrea
```

Environment

Upgrading to VKR v1.35.X.

VKS 3.6.2+v1.35 and earlier.

This issue has been resolved in VKS 3.6.3 and later.

Cause

This issue occurs due to an indeterministic selection of AddonRelease during the CNI (Antrea / Calico) upgrade phase.

When upgrading to any 1.35.x VKR, multiple Antrea or Calico AddonRelease resources may exist in the environment with the same spec.version, but will be tied to different Kubernetes minor releases (e.g., KR 1.35.2 vs 1.35.5).

In VKS versions prior to 3.6.3, the Addon Controller cannot deterministically select the correct AddonRelease that matches the target Kubernetes version.

As a result, the controller cannot resolve the exact Antrea / Calico package to deploy and the upgrade process deadlocks which causes the upgrading VKS cluster's TopologyReconcile failure.

Resolution

Upgrade the VKS supervisor service (Kubernetes service) to v3.6.3 or later which has the fix for this issue.

Workaround

If you're unable to update VKS, you can apply the fix to all clusters in a VKS namespace or to specific clusters with the attached script and the below steps:

Connect to the Supervisor cluster context
Download the addonrelease-fix.sh script attached to this KB
Create a shell script file using the contents of the addonrelease-fix.sh script
- Alternatively, upload the addonrelease-fix.sh script to the machine where you are connected to the Supervisor cluster context.
Update the addonrelease-fix.sh script to be executable:
```
chmod +x addonrelease-fix.sh
```
Validate that a dry-run of the script completes without errors and shows that it will create multiple addonInstalls:
1. If you want to run the fix against all namespaces in your Supervisor cluster, run the below dry-run command:
```
KUBECONFIG=/path/to/admin.conf ./addonrelease-fix.sh --dry-run
```
2. For a specific namespace, use the following dry-run command:
```
./addonrelease-fix.sh -n <affected VKS cluster namespace> --dry-run
```
Run the script to create the correct addonInstalls for resolving this issue:
1. To run the script against all namespaces in your Supervisor cluster, run the below command:
```
KUBECONFIG=/path/to/admin.conf ./addonrelease-fix.sh
```
2. For a specific namespace, run the following command:
```
./addonrelease-fix.sh -n <affected VKS cluster namespace>
```
Confirm that multiple addonInstalls were created prefixed with "kb-".
```
kubectl get addoninstall -A | grep "kb-"
```
There will be one newly created addonInstall for each v1.35.x VKR version in the environment.

Ensure that the affected VKS cluster shows as Available True and no longer shows the same TopologyReconcile failed errors:

kubectl get cluster -n <affected vks cluster namespace>

kubectl describe cluster -n <affected vks cluster namespace> <affected vks cluster name> | grep -i Topology

Check that the affected VKS cluster upgrade is now progressing with upgrading its control plane nodes to the new version:
```
kubectl get cluster,kcp,md,machines -n <affected VKS cluster namespace>
```
Remember that worker nodepools will not upgrade until all control plane nodes are healthy on the new version.

Additional Information

You can further restrict the script by CNI type with the below flag examples:

For antrea only:
```
./addonrelease-fix.sh --cni antrea
```
For calico only:
```
./addonrelease-fix.sh --cni calico
```

Attachments

addonrelease-fix.sh get_app