Getting "admission webhook "capi.mutating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: Cluster and variable validation failed" when upgrading the clusterClass to the latest v3.4.0

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The clusterClass conversion was triggered along with the version upgrade of the workload cluster.
  
  
• When converting the clusterClass to the latest built-in v3.4.0 via kubectl, it errors out as below.
  
  error: clusters.cluster.x-k8s.io "<cluster-name>" could not be patched: admission webhook "capi.mutating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: Cluster and variable validation failed: [spec.topology.variables[clusterEncryptionConfigYaml]: Invalid value: "\"LS0tCmFwaVZlcnNpb246IGFwaXNlcnZlci5jb25maWcuazhzLmlvL3YxCmtpbmQ6IEVuY3J5cHRpb25Db25maWd1cmF0aW9uCnJlc291cmNlczoKICAtIHJlc291cmNlczoKICAgIC0gc2VjcmV0cwogICAgcHJvdmlkZXJzOgogICAgLSBhZXNjYmM6CiAgICAgICAga2V5czoKICAgICAgICAtIG5hbWU6IGtleTEKICAgICAgICAgIHNlY3JldDogWG1SNktOTnRHa0J1N1phQ0dFTVJyNFRVbU5uZkFhVk9ZWk5wVnNpeVBTcz0KICAgIC0gaWRlbnRpdHk6IHt9Cg==\"": variable is not defined, spec.topology.variables[controlPlaneCertificateRotation]: Invalid value: "{\"activate\":true,\"daysBefore\":90}": variable is not defined, spec.topology.variables[defaultStorageClass]: Invalid value: "\"vmfs\"": variable is not defined, spec.topology.variables[extensionCert]: Invalid value: "{\"contentSecret\":{\"key\":\"tls.crt\",\"name\":\"<cluster-name>-extensions-ca\"}}": variable is not defined, spec.topology.variables[ntp]: Invalid value: "\"<IP>\"": variable is not defined, spec.topology.variables[podSecurityStandard]: Invalid value: "{}": variable is not defined, spec.topology.variables[user]: Invalid value: "{\"passwordSecret\":{\"key\":\"ssh-passwordkey\",\"name\":\"<cluster-name>-ssh-password-hashed\"}
  
  
• The error above confirms that the CAPI webhook expects the variables to be defined a certain way along with their respective values.
  
  

Environment

vSphere Kubernetes Service 3.x

Cause

This is expected system behavior when editing a cluster to change the clusterClass version.

You should not manually edit the clusterClass version.

Follow the steps in the Resolution to have the system automatically update the clusterClass version.

Resolution

When you upgrade a cluster to a higher VKR version, you can remove the skip-auto-cc-rebase annotation to have the system also automatically upgrade the clusterClass to the highest version available.

NOTE: Removal of the skip-auto-cc-rebase annotation only works when starting an upgrade. It can be removed before an upgrade is started, but the clusterClass version will not change until an upgrade is started. Similarly, the clusterClass version will not change if an upgrade is already in progress.

1. Edit the cluster yaml.
   

   kubectl edit cluster <name of the cluster> -n <namespace>

2. Remove the 'kubernetes.vmware.com/skip-auto-cc-rebase' annotation.
   
   
3. Modify the desired TKR version of the cluster under spec.topology.version of the cluster yaml.
   
   
4. Save the cluster yaml to complete the cluster re-conciliation.

What if my cluster is a newly created one with a recent clusterClass version (builtin-generic-v3.2.0 and later)?
In this case the cluster won't have the 'kubernetes.vmware.com/skip-auto-cc-rebase' annotation.
Simply upgrade the Kubernetes version of the cluster to convert the clusterClass automatically.

1. Edit the cluster yaml

   kubectl edit cluster <name of the cluster> -n <namespace>

    

2. Modify the desired TKR version of the cluster under spec.topology.version of the cluster yaml.
   
   
3. Save the cluster yaml to complete the cluster re-conciliation.

Additional Information

All the possible ways to perform the clusterClass conversion are summarized here- Variable conversion for vSphere Kubernetes Service / Tanzu Kubernetes Service (VKS/TKGS) clusters

Note: VKS v3.4 introduces the recommended builtin-generic-v3.4.0 ClusterClass. Existing clusters must rebase to this Class before or during any Kubernetes v1.33+ upgrade. Older ClusterClass (tanzukubernetescluster, builtin-generic-v3.1.0, builtin-generic-v3.2.0, and builtin-generic-v3.3.0) are deprecated and will be removed in future. Refer- Introducing VMware vSphere Kubernetes Service 3.4: Extended Kubernetes Support, Istio Service Mesh, and Enhanced Multi-Cluster Management