Cluster Update from VKR 1.32.3 to 1.33.1 fails with Webhook Validation Errors

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When attempting to update a guest cluster from TKR 1.32.3 to TKR 1.33.1, the update may fail with a webhook validation error.
The failure occurs during the admission webhook validation process and prevents the cluster from completing the upgrade.

The following error is observed:

Error from server (Forbidden): error when replacing "/tmp/kubectl-edit-#####.yaml": 
admission webhook "capi.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: 
Cluster's Kubernetes version v1.33.1+vmware.1-fips does not meet the version requirements of ClusterClass 
vmware-system-####-####/builtin-generic-v3.1.0 set by annotations: 
kubernetes.vmware.com/min-version-supported and kubernetes.vmware.com/max-version-supported

Environment

Tanzu Kubernetes Grid Service

VMware vSphere Kubernetes Service

Cause

This issue occurs when the cluster is using ClusterClass builtin-generic-v3.1.0, which does not support the schema changes introduced in TKR 1.33.1.
For TKR 1.33.1, the required ClusterClass is builtin-generic-v3.4.0. The mismatch between the TKR version and the ClusterClass version triggers webhook validation errors, blocking the update.

Key points:
ClusterClass 3.1(builtin-generic-v3.1.0) from VKS 3.1 is not compatible with VKR 1.33.1 from VKS 3.4
Schema changes were introduced between ClusterClass 3.1(builtin-generic-v3.1.0) and 3.2(builtin-generic-v3.2.0), which affect compatibility.
The skip-auto-cc-rebase annotation may prevent the automatic rebasing of the ClusterClass, further contributing to the failure.

Resolution

To remediate the issue, perform the following steps:

1. Remove the skip-auto-cc-rebase annotation
    Ensure the annotation is removed to allow ClusterClass rebasing:
    "kubectl edit cluster <cluster-name> -n <namespace>"  then
   
    Remove below line:
       kubernetes.vmware.com/skip-auto-cc-rebase

2. Update the cluster version by editing the topology.version field to the appropriate version string supported by VKS v3.4.0:
    Example: v1.33.1+vmware.1-fips-vkr.2

3. Verify the ClusterClass version
    Ensure that the cluster is now aligned with builtin-generic-v3.4.0

4. Retry the upgrade
    Re-run the cluster update process to TKR 1.33.1

Additional Information

For further details, refer to Broadcom documentation on ClusterClass auto-rebasing:
^{🔗 Auto-rebasing of VKS clusters}