Kubectl returns - Couldn't get current server API group list. The server has asked for the client to provide credentials error

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

When executing a kubectl command, the error below is returned

$ kubectl get nodes
E0515 15:07:36.791886   24915 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0515 15:07:36.815599   24915 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0515 15:07:36.831702   24915 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0515 15:07:36.844320   24915 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0515 15:07:36.855427   24915 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

All users cannot access the cluster and get the same error above.

The error message is return when the kube-apiserver fails to authenticate request

 

 

 

Environment

Tanzu Kubernetes Grid Integrated
TKGI Using OIDC Provider

 

Cause

A certificate rotation was performed prior to the error.

For some reason the /var/vcap/jobs/kube-apiserver/config/oidc-ca.pem was not updated with the "pks upgrade-cluster"

To confirm:

1. Retrieve the UUID 
   • pks cluster <cluster-name>
     or
     tkgi cluster <cluster-name>
     
     
2. ssh to the master node
   • bosh -d service-instance_<UUID> ssh master/0
     
     
3. check the "Validity Not Before" date on the oidc-ca.pem. Date is prior to the certificate rotation event
   • openssl x509 -in /<path>/oidc-ca.pem -noout -text

 

Resolution

Upgrade cluster by executing 

pks upgrade-cluster <cluster-name> 

or

tkgi upgrade-cluster <cluster-name>

 

If the above does not resolve your issue then please open a ticket with TANZU Support to troubleshoot further.