Unable to run kubectl commands on some VKS clusters due to "401 Unauthorized" error

Products

VMware vSphere Kubernetes Service

Issue/Introduction

After a Supervisor upgrade, logging into the Supervisor cluster works successfully.
However, login and command execution on some guest (VKS) clusters fail with a 401 Unauthorized error.
The Supervisor cluster runs a combination of VKS clusters managed by standard Cluster objects and legacy TanzuKubernetesCluster (TKC) objects, as viewed by running:
```
kubectl get cluster
kubectl get tkc
```

On the VKS clusters where authentication is failing, checking the logs of the guest-cluster-auth-svc pods shows that the loaded configuration contains the UID of the TKC object instead of the newer Cluster object UID.

kubectl logs -n vmware-system-auth guest-cluster-auth-###

Example Logs:

IMMDD 00:40:14.898522       1 config.go:53] Config loaded for issuer (https://<URL>/vmware.local), client ID (vmware-tes:vc:vns:k8s:<tanzukubernetescluster object UID>)
...
EMMDD 01:12:05.434098       1 token_review_endpoint.go:94] Invalid token: failed to validate JWT
oidc: expected audience "vmware-tes:vc:vns:k8s:<tanzukubernetescluster object UID>" got ["vmware-tes:vc:vns:k8s:<cluster object UID>"]

The wcp-authproxy logs at the Supervisor level print the UID of the Cluster object instead of the TKC object:

DEBUG:vclib.token:[##] Cluster UID (<cluster object id>), server (<supervisor balancer IP>)
DEBUG:vclib.token:[##] Cluster namespace = <namespace>
...
DEBUG:vclib.token:[##] Got JWT from Token Service.
...
INFO:server:[##] "127.0.0.1" - - [DD/MMM/YYYY:01:41:01 +0000] "POST /wcp/login HTTP/1.0" 200 3142 "-" "kube-plugin-vsphere bld ### - cln ### " "user@domain.com"

Inspecting the TanzuKubernetesCluster resource metadata for the problematic cluster confirms it matches the mismatched target UID:

kubectl get cluster -n namespace clustername -oyaml 
..
apiVersion: [run.tanzu.vmware.com/v1alpha3](https://run.tanzu.vmware.com/v1alpha3)
kind: TanzuKubernetesCluster
metadata:
  name: <cluster-name>
  namespace: <namespace>
  resourceVersion: "###"
  uid: <tanzukubernetescluster object UID>

Direct access (SSH) to the guest cluster Control Plane (CP) nodes works, and running kubectl local commands directly on the Control Plane nodes functions without issue.

Environment

vSphere Supervisor
vSphere Kubernetes Service

Cause

By design, when a legacy TanzuKubernetesCluster (TKC) object is present alongside standard Cluster objects, the guest-cluster-auth-svc pods prioritize and load the configuration using the UID of the TKC object.

However, the wcp-authproxy component on the Supervisor level starts using the UID of the newer Cluster object for issuing JWTs.

This mismatch of UIDs results in an invalid token error:

Expected audience in guest-cluster-auth-svc configuration: TKC UID
Audience provided in the wcp-authproxy from Supervisor: Cluster UID

Resolution

Retire the legacy TKC object:
Retire the TanzuKubernetesCluster (TKC) object for the affected cluster using the guidelines in the official Broadcom Technical Documentation:
Retiring TKG Clusters on Supervisor
Restart the guest-cluster-auth-svc DaemonSet:

Execute a rollout restart of the authentication service within the guest cluster to force the pods to reload the configuration using the correct Cluster object UID:
```
kubectl rollout restart ds guest-cluster-auth-svc -n vmware-system-auth
```
Clear local kubeconfig entries:

On your local client workstation, remove the existing cached kubeconfig file or delete the context/credentials entries associated with the problematic cluster to ensure no stale tokens are reused.
```
rm -f ~/.kube/config
```

Re-authenticate via CLI:

Re-attempt logging into the guest cluster using the kubectl-vsphere CLI:

kubectl vsphere login --server=<Supervisor-Balancer-IP> --tanzu-kubernetes-cluster-name <Cluster-Name> --tanzu-kubernetes-cluster-namespace <Namespace>