Customer could not login to the guest clusters using kubectl-vsphere
Article ID: 405709
Updated On:
Products
Issue/Introduction
Customer's are Trying to login to their guest clusters from a jump box.
If you try to run a command you will get output similar to
kubectl get nodes -v 10
I0624 13:47:39.692588 3949 loader.go:373] Config loaded from file:
E0624 13:47:39.751879 3949 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0624 13:47:39.752004 3949 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0624 13:47:39.753055 3949 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials",
"reason": "Unauthorized",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "unknown"
}
]
},
"code": 401
}]
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Environment
vSphere Supervisor 8.0 Update 3
Cause
After renewing vCenter machine certificates , the guest-cluster-auth pods are not being updated and retain the old certificates' thumbprint
The monitor service which looks for jwks changes for updated keys does not notify wcpsvc to sync the change to the Supervisor. The update is dependent on any other infra change for Supervisor to go into configure state. So the sync can be delayed for indefinite amount of time.
Engineering are aware and a fix is scheduled be updated in a future release
Resolution
Until the the permanent fix is released, the workaround is to restart the wcp service on vcenter, This will force the Supervisor to go into configure state, and the sync to occur for the guest-cluster-auth-service pod
1. Log into the vcenter Server appliance as root
2. Restart the WCP service:
vmon-cli -r wcp
3. If the customers guest cluster is below v1.31.1 then also follow the steps in KB 370252 to update the guest-cluster-auth-service pod
4. Check that you can now log in to the guest cluster
Additional Information
also see Related KB : Cannot login to vSphere with Tanzu TKC guest cluster after renewing vCenter machine certificates with error "the server has asked for the client to provide credentials"
If the customers guest cluster is v1.31.1 or higher, they should no longer need to restart the guest-cluster-auth-service pod as it is dynamically watching for changes to the configmap and will reload the public keys.
Feedback
