ClusterClass rebase to builtin-generic-v3.4.0 post VKS upgrade

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Administrators may report that post-upgrade to VKS 3.6.2 and VKr 1.33.3 in their VCF environment, the CLUSTERCLASS for their workload clusters incorrectly reflects builtin-generic-v3.4.0 instead of the expected v3.6.0. The clusters do not automatically rebase to the latest standard ClusterClass aligned with the current VKS version. This article explains how a safety annotation affects this process and provides the steps to update the ClusterClass safely.

Cause

This is expected behavior driven by two concurrent design mechanisms in VKS:

1. Auto-Rebase Safety Annotation (Primary Cause): Workload clusters created on older versions or that have undergone previous upgrades automatically receive the kubernetes.vmware.com/skip-auto-cc-rebase: "true" annotation. As detailed in KB 411452, this acts as a safety mechanism to prevent unexpected breaking changes to variable schemas during minor upgrades. The presence of this annotation completely disables the VKS auto-rebasing feature.

2. Minimum Version Baseline: To run VKr 1.33.x nodes, the absolute minimum required ClusterClass is builtin-generic-v3.4.0 (see KB 410526). During the cluster upgrade, VKS ensures this minimum baseline is met, but the safety annotation prevents it from rebasing any higher.

Resolution

Remaining on builtin-generic-v3.4.0 is a correct and valid dynamic ClusterClass configuration and will not cause outages. However, administrators wishing to utilize the latest configuration schemas by moving to v3.6.0 have two operational paths.

Important Impact Note: Moving to v3.6.0 triggers a rolling update, which gracefully replaces VMs one by one to maintain uptime while preserving persistent data.

Option 1: Auto-Rebase During a Future Kubernetes Upgrade (Recommended) Administrators can configure the cluster to automatically translate variables and rebase the ClusterClass during the next scheduled minor VKr patch or upgrade.

1. Connect into the Supervisor cluster context.

2. Edit the target cluster manifest using the following command:
   kubectl edit cluster <cluster-name> -n <namespace>

3. Locate and delete the following annotation under metadata.annotations: kubernetes.vmware.com/skip-auto-cc-rebase: "true"

4. Update the spec.topology.version field to the new target VKr version (e.g., moving to a newer patch of 1.33).
   Note: Do not manually change the spec.topology.class field. Leave it as builtin-generic-v3.4.0.

5. Save and exit the editor. VKS will automatically rewrite the variables, bump the ClusterClass to the highest supported version, and perform a single rolling update.

Option 2: Immediate Manual Upgrade (Requires Variable Conversion) If an immediate update to the ClusterClass is required without changing the VKr version, the variables must be translated manually. You cannot simply edit the YAML and change the ClusterClass string to v3.6.0. The Kubernetes API validating webhook will reject the request because the underlying variables are still formatted for the v3.4.0 schema.

1. Download the vks-variable-convert tool from KB 404143.
   
   Note: For VKS 3.5 and later (which utilize CAPI v1beta2), you must download the executable ending in -3.5 (e.g., vks-variable-convert-linux-3.5).

2. Export the cluster's current manifest to a file:
   kubectl get cluster <cluster-name> -n <namespace> -o yaml > cluster-manifest.yaml

3. Run the tool to generate a schema patch file:
   ./vks-variable-convert-<OS>-3.5 --input-file cluster-manifest.yaml --to-clusterclass builtin-generic-v3.6.0 --output-file new-variable-patches.yaml

4. Apply the patch to the live cluster:
   kubectl patch cluster -n <namespace> <cluster-name> --type=json --patch-file new-variable-patches.yaml

5. Verify the update has started by monitoring the cluster machines using
   kubectl get machines -n <namespace>.
   You will observe a rolling update where VKS provisions new nodes with the updated schema and gracefully replaces the old nodes one by one.

(Note for VCF post-GA: As this is expected product behavior acting as a safety mechanism, there is no code "fix" required in a subsequent release. The workflows above are the standard administrative resolution).

Additional Information

• Extended Kubernetes Support: Starting with the VKS 3.4 release, Broadcom introduced Extended Kubernetes Support. VKr 1.33 is included in this lifecycle model. Due to the significant architectural changes introduced (e.g., new ClusterClasses, v1beta1/v1beta2 APIs), VKr 1.33 is provided an extended support runway to allow administrators ample time to test and migrate schemas.

• For more details on supported versions and architectural changes, refer to the vSphere Kubernetes Service Release Notes.