Unable to run kubectl command on a Supervisor cluster CPVM with "error: You must be logged in to the server (Unauthorized)"

Products

• VMware vSphere with Tanzu

Issue/Introduction

• The error below is logged on the vCenter UI, as shown in the snippet.

System error occurred on Master node with identifier 42212345678901234567890123456789. Details: Failed to sync changes: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'daemonset', '--namespace', 'vmware-system-logging', '-o', 'json']' returned non-zero exit status1.. Will beretried..

• After SSH logging into the Supervisor CPVM, kubectl command errors out as below:

root@42212345678901234567890123456789 [ ~ ]# kubectl get node
error: You must be logged in to the server (Unauthorized)

Environment

VMware vSphere with Tanzu

 

Cause

When /etc/kubernetes/admin.conf files has expired certificates data like below, it is not allowed to access the cluster using the kubectl command.

Examples:

# cat /etc/kubernetes/admin.conf

# echo <certificate-authority-data-from-above> | base64 -d | openssl x509 -noout -dates
notBefore=Jun 29 22:00:00 2021 GMT
notAfter=Jul 29 22:00:00 2023 GMT

and/or

# echo <client-certificate-data-from-above> | base64 -d | openssl x509 -noout -dates
notBefore=Jul  4 01:03:48 2022 GMT
notAfter=Jul 17 10:57:26 2023 GMT

 

Resolution

1. After SSH logging to the Supervisor CPVM, check if those certificates in admin.conf have been expired.

# cat /etc/kubernetes/admin.conf
# echo <certificate-authority-data-from-above> | base64 -d | openssl x509 -noout -dates
# echo <client-certificate-data-from-above> | base64 -d | openssl x509 -noout -dates

2. Once it has been confirmed one or both of those certificates are expired, run below command to renew the certificates

# kubeadm certs renew all

3. Confirm those certificates are now renewed.

# cat /etc/kubernetes/admin.conf
# echo <certificate-authority-data-from-above> | base64 -d | openssl x509 -noout -dates
# echo <client-certificate-data-from-above> | base64 -d | openssl x509 -noout -dates