TMC-SM auth-manager-server pods crashing with error "server to run background tasks for pinniped exited"

Products

• VMware Tanzu Mission Control Self-Managed

Issue/Introduction

TMC Self-Managed auth-manager-server pods crash with below errors:

time="2024-10-24T08:37:38Z" level=info msg="starting tracing exporter at 12345" component=metrics-tracing
time="2024-10-24T08:37:38Z" level=info msg="tracing exporter started, sampling rate: 0.1" component=metrics-tracing
time="2024-10-24T08:37:38Z" level=info msg="registered background loops to reconcile oauth client secret" component=pinniped-background-tasks I1024 08:37:38.463424 1 leaderelection.go:248] attempting to acquire leader lease tmc-local/authenticator-leader-elect...
time="2024-10-24T08:37:38Z" level=info msg="Starting Prometheus at :7777" component=metrics-tracing time="2024-10-24T08:37:43Z" level=info msg="server to run background tasks for pinniped exited" component=pinniped-background-tasks error="<nil>"

Environment

TMC-SM 1.3, 1.4

Cause

Known bug, still being investigated.

Resolution

Workaround

Manually delete the authenticator-leader lease:

# kubectl delete lease authenticator-leader-elect

After this, the auth-manager-server pods should be able to start successfully.

Fix

The fix will be delivered in a future TMC-SM release. No ETA yet.