kubectl: Unable to connect to the server: x509: certificate has expired or is not yet valid

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

Customer with APM 21.3 has just reported that when checking Kubernetes (kubectl), it is showing the below error.

Unable to connect to the server: x509: certificate has expired or is not yet valid.

Environment

Kubernetes / APM 21.3 OnPrem

Cause

On the DX APM OnPrem installation, the Kubernetes system creates an own key/certificate, and the validity is 1 year and then expires.

Resolution

If you are on K8s 1.17.9 or above, the following worked:

kubeadm alpha certs check-expiration; kubeadm alpha certs renew all

Recent versions do not require the "alpha tag" anymore. For these, just use this:

kubeadm certs check-expiration; kubeadm certs renew all

then copy /etc/kubernetes/admin.conf to your ~/.kube/config

 

In order for the cluster to actually reload the keys, after you received the following message:

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

reload the relevant services with:

kubectl -n kube-system delete pod -l 'component=kube-apiserver'
kubectl -n kube-system delete pod -l 'component=kube-controller-manager'
kubectl -n kube-system delete pod -l 'component=kube-scheduler'
kubectl -n kube-system delete pod -l 'component=etcd'