How to upgrade TKG cluster with multiple VM templates
search cancel

How to upgrade TKG cluster with multiple VM templates

book

Article ID: 313086

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:
If a user creates extra node pools with customized templates, the setting will not be retained after the cluster upgrade. All node-pool nodes will be reset with the default template or the template specified in the --vsphere-vm-template-name option of the tanzu cluster upgrade command.

Cause

The TKG cluster upgrade process only supports a single VM template per cluster. Therefore, it cannot accommodate the upgrade scenario of having multiple VM templates in a single cluster.

Resolution

In a future TKG release, the node-pool with customized template upgrade issue will be addressed

Workaround:
Below are the steps to upgrade a legacy workload cluster with a customized template using APIs.
 
  1. Upgrade Packages
Retrieve the TKR version supported for the TKG release
tanzu kubernetes-release get

Target the Management cluster and edit the workload cluster to update metadata.labels.tanzuKubernetesRelease to target TKR version
kubectl edit cluster <workload cluster name>

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
:
  labels:
    tanzuKubernetesRelease: v1.24.10---vmware.1-tkg.2

This will trigger the upgrade of tanzu packages if they are bumped in this version.
Target the workload cluster and check if packageRepository and packageinstalls are bumped and in ReconcileSucceeded status
kubectl config use-context <Workload Context>
kubectl get packageRepository -A
kubectl get packageinstalls -n tkg-system
 
  1. Upgrade Control Plane nodes
Target the Management cluster.
Create a new vspheremachinetemplate for the Workload Control Plane nodes
kubectl get vspheremachinetemplate
kubectl get vspheremachinetemplate <workload CP VM template>  -o yaml > workload-cp.yaml

Modify metadata.name and spec.template.spec.template
vi workload-cp.yaml

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: work2-control-plane-1.24.10
:
spec:
  template:
    spec:
      template: /wdc-10/vm/tkg/ubuntu-2004-kube-v1-24-10+vmware-1-tkg-1

Create the new vSphereMachineTemplate
kubectl apply workload-cp.yaml

Edit workload cluster KubeadmControlPlane to upgrade Kubernetes spec.version and change spec.machineTemplate.infrastructureRef.name reference to the new one created above.
kubectl edit kcp <workload kcp> -n <namespace>

spec
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereMachineTemplate
      name: work2-control-plane-1.24.10
      namespace: default
    metadata: {}
  replicas: 1
  rolloutStrategy:
    rollingUpdate:
      maxSurge: 1
    type: RollingUpdate
  version: v1.24.10+vmware.1

This will upgrade the Control Plane nodes. The below objects can be used to track the control plane upgrade
# From Management cluster
kubectl get kcp
kuebctl get machines

# From workload cluster
kubectl get nodes
 
  1. Upgrade nodepool
For each nodepool, create a new vSphereMachineTemplate,
kubectl get vspheremachinetemplate
kubectl get vspheremachinetemplate <workload nodepool VM template> -o yaml > workload-np.yaml

Modify .metadata.name and .spec.template.spec.template
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: work2-worker-1.24.10
:
spec:
  template:
    spec:
      template: /wdc-10-vc35/vm/tkg/ubuntu-2004-kube-v1-24-10+vmware-1-tkg-custom

Create the new vSphereMachineTemplate
kubectl apply -f workload-np.yaml

Update spec.template.spec.infrastructureRef.name and spec.template.spec.version in the machineDeployment
kubectl get md -A 

kubectl edit md <Machine Deployment> -n <namespace>

spec:
  template:
    spec:
      clusterName: work2
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: VSphereMachineTemplate
        name: work2-worker-1.24.10
      version: v1.24.10+vmware.1

This will upgrade the MachineDeployment. The below objects can be used to track the upgrade
# From Management cluster
kubectl get md
kuebctl get machines

# From workload cluster
kubectl get nodes


The "Upgrade nodepool" steps can be repeated for any additional nodepools in the cluster.

Once all nodepools have been upgraded, confirm cluster status
tanzu cluster list