When using custom machine images upgrade will fail with below error.
unable to upgrade management cluster: unable to patch kubernetes version to cluster....
...
Please upload at least one VM template to continue
failed validating template path for control plane: failed to validate template path vmware-datacenter/rhel-8-kube-v1.27.5+vmware.1, target ova versions: [v1.27.5+vmware.1-tkg.1-7b64e64c7bab9f2f75a44916bc9e2c56]: invalid template path: vmware-datacenter//rhel-8-kube-v1.27.5+vmware.1: failed to find vm object for template path vmware-datacenter//rhel-8-kube-v1.27.5+vmware.1: vm 'lvmware-datacenter/rhel-8-kube-v1.27.5+vmware.1' not found
TKGM
In the vCenter, the image might be missing.
In the process of building the custom image, a few steps may be missed.
Ensure that the template is available in vCenter with the correct path. The path and template details can be verified using Govc cli.
export GOVC_URL="https://lab1.vmware.com"
export GOVC_USERNAME='Rama'
export GOVC_PASSWORD='xyzabc'
export GOVC_INSECURE=true
govc ls
In the event that the image is available in the correct location, the next step is to verify that it has been built with all the required information
Using the vCenter UI, create the virtual machine from the template. Check the vApp details under the configure tab of the newly created virtual machine.
We found that the image had a corrupt VM template that had a VERSION property 1.26 instead of 1.27. This is failing to match in the TKR resolver controller.
In the above scenario, we suggest to start creating with 1.27 template from scratch to be safe and not try to edit the existing template.