Cluster Update from VKR 1.32 to 1.33 fails - VKR 1.33 does not meet the version requirements of ClusterClass

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When attempting to update a guest cluster from TKR 1.32.3 to TKR 1.33.1, the update may fail with a webhook validation error.
The failure occurs during the admission webhook validation process and prevents the cluster from completing the upgrade.

The following error is observed:

Error from server (Forbidden): error when replacing "/tmp/kubectl-edit-#####.yaml": 
admission webhook "capi.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: 
Cluster's Kubernetes version v1.33.1+vmware.1-fips does not meet the version requirements of ClusterClass 
vmware-system-####-####/builtin-generic-v3.1.0 set by annotations: 
kubernetes.vmware.com/min-version-supported and kubernetes.vmware.com/max-version-supported

Environment

vSphere Supervisor

vSphere Kubernetes Service 3.4 and higher

Workload cluster on ClusterClass v3.3.0 or lower

Cause

This issue occurs when the cluster is using ClusterClass builtin-generic-v3.1.0, which does not support the schema changes introduced in TKR 1.33.1.
For TKR 1.33.1, the required ClusterClass at minimum is builtin-generic-v3.4.0.

The mismatch between the TKR version and the ClusterClass version triggers webhook validation errors, blocking the update.

Key points:
ClusterClass 3.1(builtin-generic-v3.1.0) from VKS 3.1 is not compatible with VKR 1.33.1 from VKS 3.4
Schema changes were introduced between ClusterClass 3.1(builtin-generic-v3.1.0) and 3.2(builtin-generic-v3.2.0), which affect compatibility.
The skip-auto-cc-rebase annotation may prevent the automatic rebasing of the ClusterClass, further contributing to the failure.

Resolution

To remediate the issue, remove the skip-auto-cc-rebase annotation alongside performing an upgrade of the workload cluster to a valid, higher VKR version.

NOTE: Removal of the skip-auto-cc-rebase annotation only works when starting an upgrade. It can be removed before an upgrade is started, but the clusterClass version will not change until an upgrade is started. Similarly, the clusterClass version will not change if an upgrade is already in progress.

1. Connect into the Supervisor cluster context
   
   
   
2. Edit the workload cluster to be upgraded:
   

   kubectl edit cluster <cluster-name> -n <namespace>

    

3. Delete the skip-auto-cc-rebase line as per below:
   

   annotations:
       kubernetes.vmware.com/skip-auto-cc-rebase

    

4. Change the VKR version to the desired VKR version to be upgraded to, for example:
   

   version: v1.33.1+vmware.1-fips-vkr.2

    

5. Save and quit.
   
   
   
6. Confirm that the cluster shows on the latest ClusterClass available and compatible with the desired VKR version:
   

   kubectl get cluster -n <namespace>

    

7. Check that the cluster is performing its rolling redeployment of nodes:
   

   kubectl get machines,vm -o wide -n <namespace>

Additional Information

See the below Upgrade Path matrix for considerations on VKR version upgrades:

vSphere Kubernetes Release Upgrade Path Matrix

 

For further details, refer to Broadcom documentation on ClusterClass auto-rebasing:
^{🔗 Auto-rebasing of VKS clusters}

A matrix regarding clusterClass compatibility with VKR versions can be found at the bottom of the below page:

Using the Versioned ClusterClass