How to upgrade TKG cluster with multiple VM templates

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
If a user creates extra node pools with customized templates, the setting will not be retained after the cluster upgrade. All node-pool nodes will be reset with the default template or the template specified in the --vsphere-vm-template-name option of the tanzu cluster upgrade command.

Cause

The TKG cluster upgrade process only supports a single VM template per cluster. Therefore, it cannot accommodate the upgrade scenario of having multiple VM templates in a single cluster.

Resolution

In a future TKG release, the node-pool with customized template upgrade issue will be addressed

Workaround:
Below are the steps to upgrade a legacy workload cluster with a customized template using APIs.
 

1. Upgrade Packages

Retrieve the TKR version supported for the TKG release

tanzu kubernetes-release get

Target the Management cluster and edit the workload cluster to update metadata.labels.tanzuKubernetesRelease to target TKR version

kubectl edit cluster <workload cluster name>

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
:
  labels:
    tanzuKubernetesRelease: v1.24.10---vmware.1-tkg.2

This will trigger the upgrade of tanzu packages if they are bumped in this version.
Target the workload cluster and check if packageRepository and packageinstalls are bumped and in ReconcileSucceeded status

kubectl config use-context <Workload Context>
kubectl get packageRepository -A
kubectl get packageinstalls -n tkg-system

 

2. Upgrade Control Plane nodes

Target the Management cluster.
Create a new vspheremachinetemplate for the Workload Control Plane nodes

kubectl get vspheremachinetemplate
kubectl get vspheremachinetemplate <workload CP VM template>  -o yaml > workload-cp.yaml

Modify metadata.name and spec.template.spec.template

vi workload-cp.yaml

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: work2-control-plane-1.24.10
:
spec:
  template:
    spec:
      template: /wdc-10/vm/tkg/ubuntu-2004-kube-v1-24-10+vmware-1-tkg-1

Create the new vSphereMachineTemplate

kubectl apply workload-cp.yaml

Edit workload cluster KubeadmControlPlane to upgrade Kubernetes spec.version and change spec.machineTemplate.infrastructureRef.name reference to the new one created above.

kubectl edit kcp <workload kcp> -n <namespace>

spec
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereMachineTemplate
      name: work2-control-plane-1.24.10
      namespace: default
    metadata: {}
  replicas: 1
  rolloutStrategy:
    rollingUpdate:
      maxSurge: 1
    type: RollingUpdate
  version: v1.24.10+vmware.1

This will upgrade the Control Plane nodes. The below objects can be used to track the control plane upgrade

# From Management cluster
kubectl get kcp
kuebctl get machines

# From workload cluster
kubectl get nodes

 

3. Upgrade nodepool

For each nodepool, create a new vSphereMachineTemplate,

kubectl get vspheremachinetemplate
kubectl get vspheremachinetemplate <workload nodepool VM template> -o yaml > workload-np.yaml

Modify .metadata.name and .spec.template.spec.template

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: work2-worker-1.24.10
:
spec:
  template:
    spec:
      template: /wdc-10-vc35/vm/tkg/ubuntu-2004-kube-v1-24-10+vmware-1-tkg-custom

Create the new vSphereMachineTemplate

kubectl apply -f workload-np.yaml

Update spec.template.spec.infrastructureRef.name and spec.template.spec.version in the machineDeployment

kubectl get md -A 

kubectl edit md <Machine Deployment> -n <namespace>

spec:
  template:
    spec:
      clusterName: work2
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: VSphereMachineTemplate
        name: work2-worker-1.24.10
      version: v1.24.10+vmware.1

This will upgrade the MachineDeployment. The below objects can be used to track the upgrade

# From Management cluster
kubectl get md
kuebctl get machines

# From workload cluster
kubectl get nodes

The "Upgrade nodepool" steps can be repeated for any additional nodepools in the cluster.

Once all nodepools have been upgraded, confirm cluster status

tanzu cluster list