Getting ""Unable to authenticate the request" err="x509: certificate has expired or is not yet valid" errors inside the kube-api server logs.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Inside the kube-api server logs of the Supervisor or Guest Cluster, the below error message is consistently seen.
  
  1 authentication.go] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time YYYY-MM-DDT<Time>Z is after YYYY-MM-DDT<Time>Z, verifying certificate SN=<Serial Number>, SKID=, AKID=<Authority key identifier> failed: x509: certificate has expired or is not yet valid: current time YYYY-MM-DDT<Time>Z is after YYYY-MM-DDT<Time>Z]"
  
  
• The error information has the details of the expiry date, serial number, authority key identifier of the certificate but the subject key identifier detail is blank.
  
  
• None of the certificates show expired in the affected cluster. The same is validated using the command below
  
  find / -type f \( -name "*.cert" -o -name "*.crt" \) -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
  
  Detailed instructions on identifying and replacing certificates can be found here- Replace vSphere with Tanzu Supervisor Certificates 

Environment

vSphere Kubernetes Service

Cause

Some embedded certificates inside the configuration files (for example admin.conf) might not show up in the output of the "find" command that is generally used to check certificates inside the cluster. As a result, our certificate management utility/tool is not able to regenerate/replace them and thus requires manual method of getting them back in order.

Resolution

1. To list all the configuration files and get the serial number & validity from the embedded certificates from each of these files, run the below command.
   
   for conf in $(ls /etc/kubernetes/ | grep conf); do
   echo "File: $conf"
   CERT_DATA=$(grep 'client-certificate-data:' /etc/kubernetes/$conf | awk '{print $2}')
   if [ ! -z "$CERT_DATA" ]; then
   echo "$CERT_DATA" | base64 -d | openssl x509 -noout -text | grep -A 3 "Serial Number"
   echo "$CERT_DATA" | base64 -d | openssl x509 -noout -text | grep -A 2 "Validity"
   else
   echo "No client-certificate-data found"
   fi
   echo "------------------------"
   done
   
   
2. To regenerate these embedded certificates, you need to use the kubernetes in-built kubeadm utility. The exact command is below.
   
   kubeadm certs renew all