VKS Workload clusters not showing up in TMC SM UI due to stale TMC SaaS annotations

Products

• VMware Tanzu Mission Control

Issue/Introduction

VKS Workload clusters are not visible in the Tanzu Mission Control Self-Managed (TMC SM) UI.

The clusters were previously registered to TMC SaaS and have since been de-registered.

Environment

Tanzu Mission Control Self-Managed (TMC SM)

vSphere with Tanzu (VKS) Workload Clusters

Cause

The VKS Workload clusters retain stale TMC SaaS annotations, and the vmware-system-tmc namespace continues to point to TMC SaaS images despite the cluster being de-registered.

Resolution

 

1. Connect to the Supervisor cluster context.

2. Check if the workload clusters have the following annotations using the command

kubectl get cluster name_cluster -oyaml

    "run.tanzu.vmware.com/agent-uid"
    "run.tanzu.vmware.com/vmware-system-tmc-cluster-group"
    "run.tanzu.vmware.com/vmware-system-tmc-applied"
    "run.tanzu.vmware.com/vmware-system-tmc-managed"
    "run.tanzu.vmware.com/tmc-already-attached"
    "run.tanzu.vmware.com/proxy-name"
    "run.tanzu.vmware.com/image-registry"
    "run.tanzu.vmware.com/auto-scaling"
    "run.tanzu.vmware.com/auto-scaler-status"
    "run.tanzu.vmware.com/kcp-status"
    "run.tanzu.vmware.com/nodepool-status"

3. Remove the identified annotations from the workload cluster configuration.

4. Switch context to the affected Workload cluster.

5. Remove the cluster agent extensions by executing the following commands:

kubectl delete crd extensions.clusters.tmc.cloud.vmware.com
kubectl delete crd agents.clusters.tmc.cloud.vmware.com
kubectl delete crd extensionresourceowners.clusters.tmc.cloud.vmware.com
kubectl delete crd extensionintegrations.clusters.tmc.cloud.vmware.com
kubectl delete crd extensionconfigs.intents.tmc.cloud.vmware.com
kubectl delete clusterrole extension-updater-clusterrole extension-manager-role agent-updater-role vmware-system-tmc-psp-agent-restricted
kubectl delete clusterrolebinding extension-updater-clusterrolebinding extension-manager-rolebinding agent-updater-rolebinding vmware-system-tmc-psp-agent-restricted
kubectl delete psp vmware-system-tmc-agent-restricted
kubectl delete namespace vmware-system-tmc

6. Verify the workload clusters now appear in the TMC SM UI.

 

Additional Information

Ensure proper kubectl context is set before executing commands to avoid modifying unintended clusters.