Supervisor cluster's tkr-status-controller-manager pod is in CrashLoopBackOff

Products

• VMware vSphere Kubernetes Service
• vSphere with Tanzu

Issue/Introduction

When checking the log of tkr-status-controller-manager Pod, it reports following error and Panic -

INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "cluster_updates_available", "controllerGroup": "cluster.x-k8s.io", "controllerKind": "Cluster", "Cluster": {"name":"custom-cluster","namespace":"my-namespace"}, "namespace": "my-namespace", "name": "custom-class", "reconcileID": "..."}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15d2242]

Environment

vSphere with Tanzu 8.0 U3c

VKS (TKG Service) 3.1.1

Cause

The issue is due to missing run.tanzu.vmware.com/resolve-tkr annotation in both the cluster and the clusterclass.

With VKS 3.2 or older, when there is no  run.tanzu.vmware.com/resolve-tkr annotation defined in either the customclass or the cluster, this will cause the tkr-status-controller-manager to panic. 

Resolution

• As a workaround add following annotation to the customclass - 

run.tanzu.vmware.com/resolve-tkr=""

• The permanent fix has been implemented in VKS 3.3+