Clusters Disconnected After TMC Self-Managed Component Restart | No valid authentication credentials

Products

• VMware Tanzu Mission Control

Issue/Introduction

In some environments where Tanzu Mission Control (TMC) is self-managed, clusters may appear as disconnected following a full pod restart of the TMC control plane components. Affected clusters may report authentication-related errors such as:

Unauthenticated desc = No valid authentication credentials

 

This may impact both the management cluster registered with TMC and other downstream managed clusters. While the TMC self-managed instance itself remains healthy, downstream connectivity can break due to stale authentication sessions.

Environment

VMware Tanzu Mission Control

Cause

TMC relies on TLS and authentication tokens to maintain communication with agents on managed clusters. A full, non-sequenced restart of TMC pods may disrupt active sessions or token propagation. This can result in clusters being unable to re-authenticate with the control plane, leading to a disconnected status in the UI and authentication errors in pod logs.

Resolution

To restore communication and resolve the disconnected status:

1. Perform a rolling restart of all TMC deployments in the TMC namespace (commonly vmware-system-tmc, svc-tmc-xX):
   • kubectl -n <tmc-namespace> rollout restart deploy
2. Manually delete pods managed by StatefulSets or DaemonSets to ensure they are re-created:
   • kubectl -n <tmc-namespace> delete pod <pod-name>
   • (You can identify pods managed by StatefulSets or DaemonSets using kubectl get <pods/sts/ds> -o wide and matching with their controllers.)
3. Monitor cluster status in the TMC UI and logs. Clusters should begin to reconnect within a few minutes as new authentication sessions are established.