Getting "admission webhook "capi.mutating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: Cluster and variable validation failed" when upgrading the clusterClass to the latest v3.4.0

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The clusterClass conversion was triggered along with the version upgrade of the workload cluster.
  
  
• When converting the clusterClass to the latest built-in v3.4.0 via kubectl, it errors out as below.
  
  error: clusters.cluster.x-k8s.io "<cluster-name>" could not be patched: admission webhook "capi.mutating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: Cluster and variable validation failed: [spec.topology.variables[clusterEncryptionConfigYaml]: Invalid value: "\"LS0tCmFwaVZlcnNpb246IGFwaXNlcnZlci5jb25maWcuazhzLmlvL3YxCmtpbmQ6IEVuY3J5cHRpb25Db25maWd1cmF0aW9uCnJlc291cmNlczoKICAtIHJlc291cmNlczoKICAgIC0gc2VjcmV0cwogICAgcHJvdmlkZXJzOgogICAgLSBhZXNjYmM6CiAgICAgICAga2V5czoKICAgICAgICAtIG5hbWU6IGtleTEKICAgICAgICAgIHNlY3JldDogWG1SNktOTnRHa0J1N1phQ0dFTVJyNFRVbU5uZkFhVk9ZWk5wVnNpeVBTcz0KICAgIC0gaWRlbnRpdHk6IHt9Cg==\"": variable is not defined, spec.topology.variables[controlPlaneCertificateRotation]: Invalid value: "{\"activate\":true,\"daysBefore\":90}": variable is not defined, spec.topology.variables[defaultStorageClass]: Invalid value: "\"vmfs\"": variable is not defined, spec.topology.variables[extensionCert]: Invalid value: "{\"contentSecret\":{\"key\":\"tls.crt\",\"name\":\"<cluster-name>-extensions-ca\"}}": variable is not defined, spec.topology.variables[ntp]: Invalid value: "\"<IP>\"": variable is not defined, spec.topology.variables[podSecurityStandard]: Invalid value: "{}": variable is not defined, spec.topology.variables[user]: Invalid value: "{\"passwordSecret\":{\"key\":\"ssh-passwordkey\",\"name\":\"<cluster-name>-ssh-password-hashed\"}
  
  
• The error above confirms that the CAPI webhook expects the variables to be defined a certain way along with their respective values.
  
  

Environment

vSphere Kubernetes Service 3.x

Cause

Manually updating both the cluster and clusterClass version at the same time is not required and making a similar change is therefore rejected by CAPI. This is because the webhook rejects the request to update the clusterClass since upgrading the cluster along with removing the "skip-auto-cc-rebase" annotation takes care of updating both the cluster and the clusterClass to the desired and latest version respectively.

Resolution

When using the "Automatic Conversion" method of converting the clusterClass to the latest supported version, you need to take care of either one of the below.

Only take care of the cluster upgrade- The same if done in the way mentioned below shall take care of the clusterClass conversion to the latest version too. Reason- when the Kubernetes version of the cluster is updated, VKS chooses to move the cluster to the latest ClusterClass that supports the requested Kubernetes version. This behavior is turned on by default for all clusters except the ones which have the kubernetes.vmware.com/skip-auto-cc-rebase annotation set. Follow the steps below to complete the same.

1. Edit the cluster yaml.
   
   k edit cluster <name of the cluster> -n <namespace>
   
   
2. Remove the 'kubernetes.vmware.com/skip-auto-cc-rebase' annotation.
   
   
3. Modify the desired version of the cluster under spec.topology.version of the cluster yaml.
   
   
4. Save the cluster yaml to complete the cluster re-conciliation.

Upgrade the clusterClass without the cluster upgrade- Follow the steps below to complete the clusterClass conversion.

1. Edit the cluster yaml.
   
   k edit cluster <name of the cluster> -n <namespace>
   
   
2. Remove the "kubernetes.vmware.com/skip-auto-cc-rebase" annotation.
   
   
3. Update the spec.topology.class field to builtin-generic-v3.x.0, where x is the VKS minor version in use. For example, to upgrade to the latest version 3.4.0, the value of the spec.topology.class field shall be builtin-generic-v3.4.0.
   
   
4. Save the cluster yaml to complete the cluster re-conciliation.

What if my cluster is a newly created one with a recent clusterClass version (builtin-generic-v3.2.0 and later)? - In this case the cluster won't have the 'kubernetes.vmware.com/skip-auto-cc-rebase' annotation. Simply upgrade the Kubernetes version of the cluster to convert the clusterClass too.

1. Edit the cluster yaml.
   
   k edit cluster <name of the cluster> -n <namespace>


2. Modify the desired version of the cluster under spec.topology.version of the cluster yaml.
   
   
3. Save the cluster yaml to complete the cluster re-conciliation.

Additional Information

All the possible ways to perform the clusterClass conversion are summarized here- Variable conversion for vSphere Kubernetes Service / Tanzu Kubernetes Service (VKS/TKGS) clusters

Note: VKS v3.4 introduces the recommended builtin-generic-v3.4.0 ClusterClass. Existing clusters must rebase to this Class before or during any Kubernetes v1.33+ upgrade. Older ClusterClass (tanzukubernetescluster, builtin-generic-v3.1.0, builtin-generic-v3.2.0, and builtin-generic-v3.3.0) are deprecated and will be removed in future. Refer- Introducing VMware vSphere Kubernetes Service 3.4: Extended Kubernetes Support, Istio Service Mesh, and Enhanced Multi-Cluster Management