Supervisor vmware-system-tkg-controller-manager pods in CrashLoopBackOff with Exit Code 2 and Nil Pointer Dereference

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:

• Pods named vmware-system-tkg-controller-manager-#### in the vmware-system-tkg namespace remain in a CrashLoopBackOff state.
• The manager container within the pod terminates with exit code 2.
• Pod logs contain the following error signature: compatibility "msg"="Distribution version is not compatible" runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference"

Environment

• Product: VMware vSphere 7.x
• Component: vSphere with Tanzu Supervisor Cluster
• Version: vSphere Kubernetes Service 3.1.0 (or similar versions post-upgrade)

Cause

The issue occurs when the tkg-controller-manager attempts to reconcile a Tanzu Kubernetes Release (TKr) distribution version that does not have a corresponding compatibility definition in the Supervisor Cluster. The lack of a valid compatibility map causes the reconciliation logic to return a null value, leading to a nil pointer dereference panic and subsequent container crash.

Resolution

Upgrade to vCenter 8.x or later.

Workaround

1) Create a local Content Library.

2) Reference Working with Items in a Library to clone the TKr templates you you need / or the ones that are compatible with your version of the Supervisor to the newly created local content library. Consult the Product Interoperability Matrix to see what TKrs are compatible with your vCenter version.

3) Change the supervisor's content library

4) Restart the vmware-system-tkg-controller-manager pods and wait for them to go into a Ready state

Additional Information

For detailed guidance on compatibility mismatches after vCenter upgrades, see KB 323445.