Tanzu Mission Control Clusters in Disconnected State due to Cluster-Auth-Pinniped-Impersonation-Proxy Tokens Accumulation

Products

• Tanzu Mission Control
• VMware Tanzu Mission Control

Issue/Introduction

Tanzu Mission Control service cluster(s) remain stuck in Disconnected state as shown in the Tanzu Mission Control web UI.

• Tanzu Mission Control pods (namespace: vmware-system-tmc) in the affected guest cluster(s) are in CrashLoopBackOff state
  
  
• While connected to the affected guest cluster, the following kubectl command returns a count of 20k or higher:

kubectl get secrets -n vmware-system-tmc | grep impersonation | wc -l

Environment

Tanzu Mission Control services managed clusters

Kubernetes clusters running TKR v1.23 and lower*

Cause

This was a known issue looked into and addressed by Tanzu Mission Control engineering team as per the below Broadcom status page:
https://status.broadcom.com/services/vmware-cloud-services/notices/spdf3lu2i69x3wx2-vmware-tanzu-mission-control-disconnected-clusters-issue

However, clusters that have remained in this disconnected from TMC state prior to the pushed fix may still be experiencing the issue.

*This includes clusters that were upgraded to a higher TKR version despite the pinniped impersonation proxy token secret accumulation and disconnected state.

TMC Engineering team's fix pushed out pods and extension updates to clean up the tokens, but disconnected clusters may not receive the fix until the impersonation proxy secrets are cleaned up and the connection to TMC is restored.

Resolution

Please open a ticket to VMware by Broadcom support for assistance in performing the clean-up of pinniped impersonation proxy tokens.

Once the pinniped impersonation proxy token secrets are cleaned up, TMC pods should return to Running state and restore connection to Tanzu Mission Control services.

This will allow the affected guest cluster TMC extensions to update and new TMC pods will be created on the cluster.

Additional Information

In the initial investigation, it was noted that the secret generation appears to be roughly every 5 minutes which means that it would take roughly 90 days to accumulate 24k secrets.