Cannot login to vSphere with Tanzu TKC guest cluster after renewing vCenter machine certificates with error "the server has asked for the client to provide credentials"

Products

• vSphere with Tanzu
• VMware Tanzu Kubernetes Grid Service (TKGs)

Issue/Introduction

After renewing vCenter machine certificates, there is a problem connecting to a Kubernetes guest cluster. 

The following error is received from the jumpbox when trying to gain access to the guest cluster. 

<user>@tanzu-virtual-machine:/tmp$ kubectl get nodes
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Running the same command with trace option.

<user>@tanzu-virtual-machine:/tmp$ kubectl get nodes -v 10

{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0613 09:19:04.332631 3236672 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0613 09:19:04.332698 3236672 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0613 09:19:04.332877 3236672 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials",
"reason": "Unauthorized",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "unknown"
}
]
},
"code": 401
}]
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Environment

• VMware vSphere with Tanzu 7.x
• VMware vSphere with Tanzu 8.x

Cause

After renewing vCenter machine certificates , the guest-cluster-auth pods retain the old certificates' thumbprint until they are restarted. 

Resolution

This will be fixed in future releases of the guest cluster.

To workaround this, restart the Guest Cluster authentication pods in the Guest Cluster:

1. Connect to the failing cluster node via ssh and export the kubeconfig admin file by running this command: export KUBECONFIG=/etc/kubernetes/admin.conf
   
   
2. Run this command to list the auth pods: kubectl get pods -A | grep cluster-auth -w 
 

3. Delete the pod with this command:
   
   kubectl rollout restart -n vmware-system-auth daemonset.apps/guest-cluster-auth-svc
   
   
4. Wait a few moments for the pod to be recreated. It will have a new age and new suffix. 
   
   kubectl get pods -A | grep cluster-auth -w 
   
   
5. On the jump box, delete the old kubeconfig file and recreate it using kubectl vsphere login.

Additional Information

NOTE: If the above steps do not resolve the issue, the issue could occur due to the connection to wp-content, in this case follow the below KB to register the content library again. 

https://knowledge.broadcom.com/external/article/323442/unable-to-pull-images-from-vsphere-with.html

If there is an error such as "HTTP request error: cannot authenticate SSL certificate for host wp-content.broadcom.com", there is likely a security appliance that injects/intercepts the SSL certificate returned from wp-content.broadcom.com. To confirm that run below from vCneter, if your security appliance/firewall intercept/inject SSL, you will see an output like below. If in the below case, the certificate is modified.

wget https://wp-content.broadcom.com/supervisor/v1/latest/lib.json --no-check-certificate

Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:1082... connected.

WARNING: cannot verify wp-content.broadcom.com's certificate, issued by `emailAddress=<PII_REDACTED>,CN=<COMPANY_REDACTED> TLS Forward Proxy v3,OU=<OU_REDACTED>:
  Self-signed certificate encountered.
Proxy request sent, awaiting response... 503 Service Unavailable