When you run kubectl get tkr -n tkr-system or tanzu kubernetes-release get in Tanzu Kubernetes Grid 1.3, it says no resource found.
#kubectl get tkr -n tkr-system No resources found
When trying to create a workload cluster with a desired version it fails.
$ tanzu cluster create --tkr v1.19.9---vmware.2-tkg.1 -f pws0003_a1_template.yaml -v6 [2021-11-24 16:05:37.723] Error: could not find a matching TanzuKubernetesRelease for name "v1.19.9---vmware.2-tkg.1"
When we check the logs of tkr-controller-manager in the management cluster, we see events similar to the following:
#tkr-controller-manager logs
2021-12-03T08:07:59.215Z INFO failed to reconcile TKRs, retrying {"error": "failed to sync up TKRs with the BOM repository: failed to reconcile the BOM ConfigMap: failed to create BOM ConfigMap for image dev-tb-5ccmtkh9002v.5gc.home/tkg/tkr-bom:sha256-09355c8b1907b8fd0e1e3762cb36628b015f4b79f2856e8c98210419f823e0f3.imgpkg: configmaps \"v1.21.2---vmware.1-tkg.1\" already exists"}
The tkr-controller-manager code is looking for all tags from the images under tkr-bom path, and then creates configmap for them.
However, from the above output log, the configmap for v1.21.2_vmware.1-tkg.1 already exists. As a result, it cannot create the same configmap for the dup tag.
In the following screenshot, we see some additional tags with "..(2)".
Login to the Harbor GUI and remove additional tags for the bom-images under the path projects/tkg/tkr-bom.
You can click on the image name to edit a tag's name.
After removing these extra (empty) tags from the harbor registry, wait for the tkr controller to reconcile.
Check again with the command kubectl get tkr -n tkr-system or tanzu kubernetes-release get. This should give you the output and allow you to create the workload cluster with desired version of kubernetes.