Cannot login to vSphere with Tanzu TKC guest cluster after renewing vCenter machine certificates with error "the server has asked for the client to provide credentials"

Products

• vSphere with Tanzu
• VMware Tanzu Kubernetes Grid Service (TKGs)

Issue/Introduction

After renewing vCenter machine certificates, there is a problem connecting to a Kubernetes guest cluster. 

The following error is received from the jumpbox when trying to gain access to the guest cluster. 

<user>@tanzu-virtual-machine:/tmp$ kubectl get nodes
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Running the same command with trace option.

<user>@tanzu-virtual-machine:/tmp$ kubectl get nodes -v 10

I0613 09:19:04.308398 3236672 loader.go:373] Config loaded from file: /home/<user>/.kube/config
I0613 09:19:04.309104 3236672 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.26.4+vmware.wcp.0 (linux/amd64) kubernetes/c3baec6" -H "Authorization: Bearer <masked>" 'https://10.x.x.3:6443/api?timeout=32s'
...
I0613 09:19:04.321473 3236672 request.go:1171] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0613 09:19:04.321708 3236672 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0613 09:19:04.321724 3236672 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0613 09:19:04.321798 3236672 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.26.4+vmware.wcp.0 (linux/amd64) kubernetes/c3baec6" -H "Authorization: Bearer <masked>" 'https://10.x.x.3:6443/api?timeout=32s'
I0613 09:19:04.323414 3236672 round_trippers.go:553] GET https://10.x.x.3:6443/api?timeout=32s 401 Unauthorized in 1 milliseconds
...
I0613 09:19:04.323899 3236672 request.go:1171] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0613 09:19:04.324096 3236672 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
...
I0613 09:19:04.326028 3236672 round_trippers.go:553] GET https://10.x.x.3:6443/api?timeout=32s 401 Unauthorized in 1 milliseconds
I0613 09:19:04.326092 3236672 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 1 ms Duration 1 ms
...
0613 09:19:04.326563 3236672 request.go:1171] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0613 09:19:04.326761 3236672 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
...
I0613 09:19:04.332442 3236672 request.go:1171] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0613 09:19:04.332631 3236672 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0613 09:19:04.332698 3236672 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0613 09:19:04.332877 3236672 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials",
"reason": "Unauthorized",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "unknown"
}
]
},
"code": 401
}]
error: You must be logged in to the server (the server has asked for the client to provide credentials)

 

 

Environment

VMware vSphere with Tanzu 7.x

VMware vSphere with Tanzu 8.x

Cause

After renewing vCenter machine certificates, the guest-cluster-auth pods retain the old certificates' thumbprint until they are restarted. 

Resolution

This will be fixed in future releases of the guest cluster.

 

To workaround this, restart the Guest Cluster authentication pods.

1. Connect to the failing cluster node via ssh and export the kubeconfig admin file. 
   
   kubectl get pods -A | grep cluster-auth -w 

2. Delete the pod. 
   
   kubectl delete pod -n vmware-system-auth guest-cluster-auth-svc-xxxx
   
   
3. Wait a few moments for the pod to be recreated. It will have a new age and new suffix. 
   
   kubectl get pods -A | grep cluster-auth -w 
   
   
4. On the jump box, delete the old kubeconfig file and recreate it using kubectl vsphere login.