Errors encountered when attempting to gather admin-kubeconfig from tanzu mission-control CLI

Products

• VMware Tanzu Mission Control
• vSphere with Tanzu

Issue/Introduction

When running tanzu mission-control cluster admin-kubeconfig get <CLUSTER_NAME> -p <PROVISIONER_NAME> -m <MANAGEMENT_CLUSTER_NAME>, users might see the following errors returned:

Scenario 1:

• Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
• This failure impacts all Guest Clusters managed by a single Supervisor Cluster.
• Gathering the admin-kubeconfig from the TMC GUI works without issue.

Scenario 2:

• RPC errors over 40% of the time.
  • Error: rpc error: code = NotFound desc = (target=mc:<id>, intentId=<id>): AdminKubeconfig.run.tanzu.vmware.com "<>" not found
  • Using the attached script you see the following in the header:

    • 2025-01-07 00:47:00 - Retrieving kubeconfig for cluster: <cluster>

    • Cluster admin kubeconfig for <cluster>:

    • HTTP/1.1 404 Not Found

• This failure impacts all Guest Clusters managed by a single Supervisor Cluster.

 

Environment

vSphere with Tanzu Supervisor Cluster managed by Tanzu Mission Control SaaS.

 

Cause

This problem is caused when the Supervisor Cluster is unreachable, or when certain TMC pods on the Supervisor Cluster are degraded.

Resolution

Scenario 1:

Restart the TMC intent-agent pod on the Supervisor Cluster. If this doesn't resolve the issue, follow the steps in the Additional Information field to gather more verbose messaging on the failure.

Scenario 2: 

Restart the TMC sync-agent and vsphere-resource-retriever pods on the Supervisor Cluster. The vsphere-resource-retriever generates the adminkubeconfig and then sync-agent sents it back to TMC.

Additional Information

Use the script attached to this KB to send a curl request to the TMC API for more verbose messaging on the failure.

 

• In order to run the script, to set your TMC API token as an environmental variable:
  
  export VMW_CLOUD_API_TOKEN=<YOUR_API_TOKEN>
  
  
• Next, edit the script and enter your specific environmental variables for the cluster you wish to test against:
  
  
  management_cluster="MANAGEMENT_CLUSTER_NAME"  # Replace with your management cluster name

  provisioner="GUEST_CLUSTER_NAMESPACE"  # Replace with your provisioner name

  cluster_name="GUEST_CLUSTER_NAME"  # Replace with your cluster name

  api_host="https://<API_HOST>.tmc.cloud.vmware.com"  # Replace if your API host is different
  
  
• Finally, chmod +x the script to make it executable:
  
  chmod +x <SCRIPT_NAME>