TKGm pod tanzu-addon-controller-manager in CrashLoopBackOff - no matches for kind "Cluster"

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

Symptoms:
TKGm Management cluster creation fails and tanzu-addon-controller-manager pod is in CrashLoopBackOff state.
Logs from the pod show that the "Cluster" CRD has not been created on the cluster.

controller-runtime/source "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"Cluster\" in version \"cluster.x-k8s.io/v1alpha3\""

Presence of the Cluster API CRD can be verified with:
kubectl api-resources | grep cluster.x-k8s.io
 

Environment

VMware Tanzu Kubernetes Grid Plus 1.x
VMware Tanzu Kubernetes Grid 1.x

Cause

The tkr-controller-manager pod downloads the BOM files from the from the VMware registry or private registry for air-gapped environment. These BOM files contain the Cluster API CRDs.

Review the tkr-controller-manager pod logs and check for the presence of the error below:

{"error": "failed to sync up TKRs with the BOM repository: failed to reconcile the BOM ConfigMap: failed to list current available BOM image tags: Get \"https://REGISTRY-FQDN": dial tcp REGISTRY-IP:443: i/o timeout"}

Resolution

Verify connectivity to the registry 
kubectl get pods -n tkr-system -o wide
kubectl get nodes -o wide
ssh capv@NODE-IP
curl -kv https://REGISTRY-FQDN:443

Verify that you can download images from the registry
crictl pull REGISTRY-FQDN/IMAGE-PATH

If the above verification steps are successful, check for the known issue described in https://kb.vmware.com/s/article/86496.

Apply the workaround in this KB to the Worker VM and restart the tkr-controller-manager pod.

To apply the workaround at cluster creation time, copy the contents below in a file and save in ~/.tanzu/tkg/providers/infrastructure-vsphere/ytt.
See documentation for further details on overlays as the directory is different for different TKG versions.

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"})
---
spec:
  kubeadmConfigSpec:
    preKubeadmCommands:
      #@overlay/append
      - ethtool -K eth0 tx-udp_tnl-segmentation off && ethtool -K eth0 tx-udp_tnl-csum-segmentation off && ethtool -K eth0 tx off && ethtool -K eth0 rx off

#@overlay/match by=overlay.subset({"kind": "KubeadmConfigTemplate"})
---
spec:
  template:
    spec:
      preKubeadmCommands:
      #@overlay/append
      - ethtool -K eth0 tx-udp_tnl-segmentation off && ethtool -K eth0 tx-udp_tnl-csum-segmentation off && ethtool -K eth0 tx off && ethtool -K eth0 rx off