error: You must be logged in to the server (Unauthorized) after retiring a Tanzu Kubernetes Cluster (TKC)

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After following the instructions in Retire TanzuKubernetesCluster Resources, you are no longer able to authenticate against the retired cluster and receive the message: error: You must be logged in to the server (Unauthorized) when running kubectl commands.

• Issue may be intermittent
• May occur after deleting your Kubeconfig

Environment

The Kubernetes Releases (KR) are affected:

• v1.23.15---vmware.1-tkg.4
• v1.23.8---vmware.2-tkg.2-zshippable
• v1.24.9---vmware.1-tkg.4
• v1.25.7---vmware.3-fips.1-tkg.1
• v1.26.13---vmware.1-fips.1-tkg.3
• v1.26.5---vmware.2-fips.1-tkg.1
• v1.27.10---vmware.1-fips.1-tkg.1
• v1.27.11---vmware.1-fips.1-tkg.2
• v1.27.16---vmware.2-fips-vkr.2
• v1.28.15---vmware.3-fips-vkr.3
• v1.28.8---vmware.1-fips.1-tkg.2
• v1.29.15---vmware.1-fips-vkr.2
• v1.29.4---vmware.3-fips.1-tkg.1
• v1.30.1---vmware.1-fips-tkg.5
• v1.30.11---vmware.1-fips-vkr.2
• v1.30.14---vmware.1-fips-vkr.3
• v1.30.8---vmware.1-fips-vkr.1

Cause

The guest-cluster-auth-svc may not detect that the TKC has been retired and fails to restart automatically to pick up the changes.

Resolution

1) Log into the supervisor context using kubectl

2) Get the cluster's kubeconfig file:

• export CLUSTER_NAME=<YOUR KUBERNETES CLUSTER NAME> NS=<CLUSTER'S NAMESPACE>
  
  
• kubectl get secret  -n ${NS} ${CLUSTER_NAME}-kubeconfig -o jsonpath={.data.value} | base64 -d > ${CLUSTER_NAME}-kubeconfig
  
  
• KUBECONFIG=${CLUSTER_NAME-kubeconfig kubectl get node
  
  

3) Retrieve the guest-cluster-aut-svc pods:

${KUBECONFIG} kubectl get pods -n vmware-system-auth -l name=guest-cluster-auth-svc

NAME                           READY   STATUS    RESTARTS       AGE

guest-cluster-auth-svc-h9pww   1/1     Running   1 (46s ago)    67m

guest-cluster-auth-svc-kjhfm   1/1     Running   1 (96s ago)    69m

guest-cluster-auth-svc-sqlft   1/1     Running   1 (108s ago)   71m

4) Delete the guest-cluster-aut-svc pods so they will automatically get re-created:

${KUBECONFIG} kubectl delete pods -n vmware-system-auth -l name=guest-cluster-auth-svc