Variable conversion for vSphere Kubernetes Service / Tanzu Kubernetes Service (VKS/TKGS) clusters

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Conversion of cluster variables to the new variable schema introduced in VKS 3.2.0

Environment

vSphere Kubernetes Service version 3.2.0 and above

Resolution

Table of Contents

• Overview
• Automatic conversion
  • Steps to convert the variables automatically
• Manual conversion
  • NTP settings for the cluster
  • Storage classes for use in the cluster
• Manual conversion using the vks-variable-convert tool

Overview

The following covers the process for automated as well as manual conversion of cluster variables using the tanzukubernetescluster or builtin-generic-v3.1.0 ClusterClass. This article contains the information which will allow cluster operators to:

1. Make changes to clusters using the ClusterClasses mentioned above to adopt the new variable schema
2. Create new clusters using the newer ClusterClass versions starting VKS 3.2.0 by mapping the older variable values to the new schema

With the release of vSphere Kubernetes Service v3.2.0, the variables for customizing the clusters using the v1beta1 API have undergone significant changes. The variable schema on the new ClusterClass versions introduce a hierarchical variable structure. This new variable schema groups multiple configurations under top level variables such that similar customization variables can be set together under one variable. For example, instead of having three different variables for proxy setting, CA trust configurations and NTP servers, the new variable schema exposes an osConfiguration variable which contains these three individual settings.

Note: Any changes to the Kubernetes version or the ClusterClass versions and/or variables will likely result in a rollout of the nodes of the cluster. 

Automatic conversion

TKGS/VKS exposes the concept of auto-rebasing clusters to use the latest supported ClusterClass. When the Kubernetes version of the cluster is updated, VKS chooses to move the cluster to the latest ClusterClass that supports the requested Kubernetes version. This behavior is turned on by default for all clusters except the ones which have the kubernetes.vmware.com/skip-auto-cc-rebase annotation set.

Any existing clusters created using a VKS version less than v.3.2.0 will have this skip rebase annotation set on them by default after upgrade to VKS version 3.2.0 or above. Inspect the metadata.annotations field of the cluster object after the VKS version upgrade to confirm the same.

Steps to convert the variables automatically

1. Edit the cluster manifest using kubectl edit cluster -n <namespace> <cluster-name>

2. Remove the kubernetes.vmware.com/skip-auto-cc-rebase annotation from the metadata.annotations field.

3. Update the spec.topology.version field to a newer minor/patch. Skip step 4 if updating the Kubernetes version of the cluster. Otherwise go to step 4 if Kubernetes version upgrade is skipped.

4. (Optional) Update the spec.topology.class field to builtin-generic-v3.x.0, where x is the VKS minor version in use. 

5. Save the changes and close the editor.

This should cause the variables to be automatically updated to the new variable schema. Inspect the variables on the cluster using the following command. It makes a check for the presence key extensionCert, which has been removed from the new variable schema and instead handled internally via VKS.

# This should return no output since the variable is removed
kubectl get cluster -n <NAMESPACE> <CLUSTER> -o jsonpath='{.spec.topology.variables}' | grep extensionCert

Note: You can use any of the ClusterClass versioned support by VKS but VKS will always rebase the Cluster to the latest ClusterClass supporting the Kubernetes version. Please consult the release notes of the VKS version in use for more information about Kubernetes versions supported by a ClusterClass.

Manual conversion

If for any reason you do not want to leverage the automatic rebasing, you can use this table to interpret the destination variables for all the variables in the new schema. The table is a subset of the original table published in ClusterClass Variable Conversion to showcase the manual conversion process:

Variable for TKG Service v3.1.x or earlier	New variable for TKG Service v3.2.x or later
ntp	osConfiguration.ntp.servers
storageClass	storageClass
storageClasses	vsphereOptions.persistentVolumes.availableStorageClasses

NTP settings for the cluster

If the existing cluster YAML has the ntp variable with a value defined, this value now should be included the osConfiguration variable under the ntp section.

# original variable declaration
- name: ntp
  value: ntp.example.com
# new variable declaration
- name: osConfiguration
  value:
    ntp:
      servers:
      - ntp.example.com

Storage classes for use in the cluster

Similarly, for the storageClasses variable, this should now reside under the vsphereOptions variable under the section persistentVolumes against the availableStorageClasses sub-section.

# original variable declaration
- name: storageClasses
  value:
  - storage-class-1
  - storage-class-2
# new variable declaration
- name: vsphereOptions
  value:
    persistentVolumes:
      availableStorageClasses:
      - storage-class-1
      - storage-class-2

Use the same method, to create a new set variables and replace it under the spec.topology.variables field of the cluster object. Any variables overridden at the control plane or node pool level should also be transformed using the same process.

Manual conversion using the vks-variable-convert tool

As shown above, the process to initiate a manual variable conversion can be time-consuming, error prone requires knowledge about the new variable schema. Instead, you can use the attached vks-variable-convert tool to convert the cluster YAML using the old variable schema to the new schema.

1. Download and install the tool at a location of your choosing.
2. Download the YAML output of the Cluster object to a file using kubectl get cluster -n  <namespace> <cluster-name> -o yaml > cluster-manifest.yaml 
3. Provide the file and the VKS ClusterClass you want to use as inputs to the tool:
   ./vks-variable-convert --input-file cluster-manifest.yaml --to-clusterclass builtin-generic-v3.2.0 --output-file new-variable-patches.yaml 
4. Follow the on-screen instructions to view the new cluster object output locally and patch the cluster once the information is verified.

Example usage

Retrieve the cluster manifest for the cluster you want to convert to the new schema

kubectl get cluster <CLUSTER> -o yaml > cluster-manifest.yaml

Run the tool 

./vks-variable-convert --input-file cluster-manifest.yaml --to-clusterclass builtin-generic-v3.2.0 --output-file new-variable-patches.yaml

Run the following command to verify the output locally

kubectl patch -f cluster-manifest.yaml --type=json --patch-file new-variable-patches.yaml --local -o yaml

Run the following command to verify output of the cluster with converted cluster variables

kubectl patch cluster -n <NAMESPACE> <CLUSTER> --type=json --patch-file new-variable-patches.yaml --dry-run=client -o yaml

Run the following command to patch the converted cluster variables

kubectl patch cluster -n <NAMESPACE> <CLUSTER> --type=json --patch-file new-variable-patches.yaml

Additional Information

1. The variable definitions for the older or existing variables can be found at Cluster v1beta1 API

2. The variable definitions for the newer variable can be found at ClusterClass Variables for Customizing a Cluster