cannot login into gust cluster: couldn't get current server API group list: the server has asked for the client to provide credentials

Products

• VMware vSphere with Tanzu

Issue/Introduction

In Tanzu environment, after login into guest cluster you can face the following error:

$ kubectl get pods
E0729 13:36:48.454594  597605 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0729 13:36:48.457347  597605 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0729 13:36:48.460000  597605 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0729 13:36:48.463042  597605 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0729 13:36:48.465811  597605 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Cause

Most probably reason is lack of synchronization between secrets and guest cluster login.

Resolution

Restart vCenter and all cluster nodes, following the order 

Shutdown Order	Component
1	Containerized customer workloads
2	Find out the location of the vSphere with Tanzu virtual machines *
3	vSphere Cluster Services virtual machines in the VI workload domain *
4	vCenter Server for the VI workload domain *
5	Supervisor Cluster Control Plane virtual machines
6	Tanzu Kubernetes cluster control plane virtual machines
7	Tanzu Kubernetes cluster worker virtual machines
8	Harbor virtual machines
9	NSX Edge nodes in the VI workload domain *
10	NSX Manager nodes for the VI workload domain *
11	vSAN and ESXi hosts in the VI workload domain *

 

Have a look at this document for detailed information: https://docs.vmware.com/en/VMware-Cloud-Foundation/4.5/vcf-operations/GUID-38BA4D29-A7B0-4E52-9D83-F65AFD481465.html