TKGm Management Cluster upgrade fails with error "VSphereMachineTemplate spec.template.spec field is immutable. Please create a new resource instead."

Products

• VMware Tanzu Kubernetes Grid Plus
• VMware Tanzu Kubernetes Grid Plus 1.x
• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

Management Cluster (MC) upgrade fails with below error:

Installing the default ClusterClass...
Error: unable to install the default ClusterClass: failed to apply /home/<user-name>/.config/tanzu/tkg/clusterclassconfigs/tkg-vsphere-default-vX.X.X.yaml: kubectl apply failed

Error from server (Invalid): error when applying patch:
....
to:
Resource: "infrastructure.cluster.x-k8s.io/v1beta1, Resource=vspheremachinetemplates", GroupVersionKind: "infrastructure.cluster.x-k8s.io/v1beta1, Kind=VSphereMachineTemplate"
Name: "tkg-vsphere-default-vX.X.X-control-plane", Namespace: "tkg-system"
....
to:
Resource: "infrastructure.cluster.x-k8s.io/v1beta1, Resource=vspheremachinetemplates", GroupVersionKind: "infrastructure.cluster.x-k8s.io/v1beta1, Kind=VSphereMachineTemplate"
Name: "tkg-vsphere-default-vX.X.X-worker", Namespace: "tkg-system"
....
VSphereMachineTemplate spec.template.spec field is immutable. Please create a new resource instead.

Cause

The issue occurs when there's a mismatch between the .spec.template.spec field in /home/<user-name>/.config/tanzu/tkg/clusterclassconfigs/tkg-vsphere-default-vX.X.X.yaml and the "tkg-vsphere-default-vX.X.X-control-plane" and "tkg-vsphere-default-vX.X.X-worker" VSphereMachineTemplate objects in the cluster.

Resolution

1. List the VSphereMachineTemplate objects in the tkg-system namespace in your Management context:
   # kubectl get vspheremachinetemplate -n tkg-system
   
   For example:
   ubuntu@jumpbox:~$ kubectl get vspheremachinetemplate -n tkg-system
NAME                                       AGE
mgmt-slot34rp35-control-plane-5fqnv        50d
mgmt-slot34rp35-md-0-infra-x2lkw           50d
tkg-vsphere-default-v1.1.1-control-plane   28h
tkg-vsphere-default-v1.1.1-worker          50d
   
   
2. Examine the .spec.template.spec field in "tkg-vsphere-default-vX.X.X-control-plane" and "tkg-vsphere-default-vX.X.X-worker" VSphereMachineTemplate objects:
   # kubectl get vspheremachinetemplate -n tkg-system tkg-vsphere-default-vX.X.X-control-plane -o yaml
# kubectl get vspheremachinetemplate -n tkg-system tkg-vsphere-default-vX.X.X-worker -o yaml
   
   For example:
   ubuntu@jumpbox:~$ kubectl get vspheremachinetemplate -n tkg-system tkg-vsphere-default-v1.1.1-control-plane -o yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"infrastructure.cluster.x-k8s.io/v1beta1","kind":"VSphereMachineTemplate","metadata":{"annotations":{},"creationTimestamp":"2024-06-26T15:53:12Z","generation":1,"name":"tkg-vsphere-default-v1.1.1-control-plane","namespace":"tkg-system","ownerReferences":[{"apiVersion":"cluster.x-k8s.io/v1beta1","kind":"ClusterClass","name":"tkg-vsphere-default-v1.1.1","uid":"<uid>"}],"resourceVersion":"4874","uid":"<uid>"},"spec":{"template":{"spec":{"cloneMode":"fullClone","datacenter":"/<datacenter-name>","datastore":"/<datastore-path>","diskGiB":40,"folder":"/<folder-path>","memoryMiB":8192,"network":{"devices":[{"dhcp4":true,"networkName":"/<network-name>"}]},"numCPUs":4,"resourcePool":"/<resourcePool-path>","server":"<server>","storagePolicyName":"","template":" "}}}}
  creationTimestamp: "2024-08-15T10:00:01Z"
  generation: 1
  name: tkg-vsphere-default-v1.1.1-control-plane
  namespace: tkg-system
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: ClusterClass
    name: tkg-vsphere-default-v1.1.1
    uid: <uid>
  resourceVersion: "23975096"
  uid: <uid>
spec:
  template:
    spec:
      cloneMode: fullClone
      datacenter: /<datacenter-name>
      datastore: /<datastore-path>
      diskGiB: 40
      folder: /<folder-path>
      memoryMiB: 8192
      network:
        devices:
        - dhcp4: true
          networkName: /<network-name>
      numCPUs: 4
      resourcePool: /<resourcePool-path>
      server: <server>
      storagePolicyName: ""
      template: ' '
   
   
3. Compare the .spec.template.spec field of the "tkg-vsphere-default-vX.X.X-control-plane" and "tkg-vsphere-default-vX.X.X-worker" VSphereMachineTemplate sections in /home/<user-name>/.config/tanzu/tkg/clusterclassconfigs/tkg-vsphere-default-vX.X.X.yaml file with the above.
   Look for any differences.
   
   
4. /home/<user-name>/.config/tanzu/tkg/clusterclassconfigs/tkg-vsphere-default-vX.X.X.yaml contents come from secret "tkg-pkg-tkg-system-values" in the tkg-system namespace.
   Write the values to a file:
   # kubectl get secret -n tkg-system tkg-pkg-tkg-system-values -o jsonpath='{.data.tkgpackagevalues\.yaml}' | base64 -d > tkgpackagevalues.yaml
   
   
5. Examine tkgpackagevalues.yaml file and look for the differences you located in step #3.
   For example, if you noticed .spec.template.spec.resourcePool is different, this field corresponds to "VSPHERE_RESOURCE_POOL" in tkgpackagevalues.yaml file.
   
   
6. Once you've located the wrong values, edit the tkgpackagevalues.yaml file with the correct ones:
   # vim tkgpackagevalues.yaml
   
   
7. Patch the "tkg-pkg-tkg-system-values" secret with the new values:
   # kubectl patch secret -n tkg-system tkg-pkg-tkg-system-values --patch "{\"data\":{\"tkgpackagevalues.yaml\":\"$(base64 -w 0 tkgpackagevalues.yaml)\"}}"
   
   
8. Verify that the new values have been correctly patched:
   # kubectl get secret -n tkg-system tkg-pkg-tkg-system-values -o jsonpath='{.data.tkgpackagevalues\.yaml}' | base64 -d

Next time you trigger the upgrade you shouldn't see the original error messages again as /home/<user-name>/.config/tanzu/tkg/clusterclassconfigs/tkg-vsphere-default-vX.X.X.yaml file will have been updated with the new expected values.