Tanzu Mission Control (TMC) Workload Clusters in Unknown State due to Unauthenticated Errors

Products

• Tanzu Kubernetes Runtime
• VMware Tanzu Mission Control - SM
• VMware Tanzu Mission Control

Issue/Introduction

In the Tanzu Mission Control (TMC) web UI, one or more workload clusters with provider vSphere with Tanzu (also known as vSphere Supervisor) remain stuck in Unknown state.

 

While connected to the workload cluster context, one or more of the following symptoms are observed:

• The TMC system pods are present under the vmware-system-tmc namespace:
  

  kubectl get pods -n vmware-system-tmc

• Depending on the scenario, these TMC system pods may be unhealthy or Running state.
  
  
  
• When viewing the logs for intent-agent in vmware-system-tmc, errors similar to the following are present:
  

  kubectl logs -n vmware-system-tmc <intent-agent pod>
  
  unauthenticated - no valid authentication credentials

Environment

vSphere Supervisor managed by Tanzu Mission Control (TMC)

Cause

Tanzu Mission Control (TMC) monitors the status of a workload cluster by deploying pods and services directly into the cluster.

If these pods are failing or not working properly, TMC will not be able to accurately report on the state of the cluster.

Resolution

Confirm that TMC pods are not failing in the workload cluster.

1. Connect into the affected workload cluster's context
   
   
2. Check the list of all TMC pods to confirm on their current state:
   

   kubectl get pods -n vmware-system-tmc

   If any TMC pods are failing, check their logs and describe the pods for more information on the cause. All TMC pods must be in a healthy state for accurate monitoring.

   kubectl describe pod -n vmware-system-tmc <pod name>
   
   kubectl logs -n vmware-system-tmc <pod name>

    

3. With tanzu CLI and tanzu mission control plugin available, run the below validate command against the workload cluster and its kubeconfig:
   

   tanzu mission-control cluster validate --kubeconfig <path/to/kubeconfig for the workload cluster>

   
   This command can also be found in the Tanzu Mission Control (TMC) web UI, after navigating to the affected workload cluster which should have a tanzu cli command to validate the workload cluster with the tanzu mission control plugin.
   
   
4. If successful, the validate command will report information on the TMC pods and overall cluster health.
   
   
5. If the validate command outputs (at the top) an error message similar to the following:
   

   Checks:
      Connectivity to TMC     Invalid authentication credentials: rpc error: code - Unauthenticated desc - No valid authentication credentials

   
   Perform the reattach command suggested in its output:
   

   Reattach this cluster to VMware Tanzu Mission Control using this command:
   tanzu mission-control cluster reattach <workload cluster name> --management-cluster-name <management cluster name> --provisioner-name <namespace> --kubeconfig </path/to/kubeconfig>
   

    

6. Once the reattach command completes, verify in Tanzu Mission Control (TMC) web UI that the monitoring is reporting accurately.

Additional Information

TMC CLI was deprecated in 2023 and will not function properly with TMC.

Tanzu CLI with TMC plugin should be used instead.

Download and Install the Tanzu CLI and Tanzu Mission Control Plug-ins

Ensure that all plugins are updated properly with the below command:

tanzu plugin sync --target tmc