Getting error "you must be logged in to the server (the server has asked for the client to provide credentials)" when trying to run kubectl commands on the supervisor cluster.

Products

• vSphere with Tanzu

Issue/Introduction

You encounter a situation where,

1. On running any kubectl command on the supervisor cluster, it errors out with- "you must be logged in to the server (the server has asked for the client to provide credentials".
2. The kubelet service, kube-api server, etcd and all the other static containers are running.
3. The certificates on the supervisor cluster are all up to date.
4. The certificate inside the "client-certificate-data" of the admin.conf file is expired.

Environment

vSphere with Tanzu

Cause

During cluster creation, the file admin.conf is created by kubeadm on control plane nodes and it contains a certificate with Subject: O = kubeadm:cluster-admins, CN = kubernetes-admin. The same is renewed/replaced via kubeadm itself and is not renewed/replaced by the certmgr script used to replace other supervisor certificates.

Resolution

To renew the admin.conf certificate, run the command- "kubeadm certs renew all". If run successfully, it explicitly outputs that the certificate embedded in the kubeconfig file is renewed. For example, see below

 

root@<supervisor-dns-name> [ /etc/kubernetes ]# kubeadm certs renew all

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
kubeConfig file /etc/kubernetes/scheduler.conf does not have an embedded client certificate
To see the stack trace of this error execute with --v=5 or higher

 

Once completed also run through the knowledge article Unable to run kubectl commands in a Supervisor Cluster due to "error: You must be logged in to the server (Unauthorized)" to complete the restart of related pods/containers.

 

NOTE: As this does not renew all certificates in the system also run through knowledge article Replace vSphere with Tanzu Supervisor Certificates to make sure any other expired certificates are also updated.

Additional Information

Certificate Management with kubeadm: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/