Error: failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Kubelet is failing to start
• In TCA-CP, the command openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate returns notAfter=Nov 17 15:32:57 2024 GMT
• journalctl displays the following errors:

Nov 21 09:42:18 ##.##.## kubelet[1354093]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Nov 21 09:42:18 ##.##.## kubelet[1354093]: I1121 09:42:18.205989 1354093 server.go:412] "Kubelet version" kubeletVersion="v1.26.5+vmware.2"
Nov 21 09:42:18 ##.##.## kubelet[1354093]: I1121 09:42:18.206053 1354093 server.go:414] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 21 09:42:18 ##.##.## kubelet[1354093]: I1121 09:42:18.206557 1354093 server.go:836] "Client rotation is on, will bootstrap in background"
Nov 21 09:42:18 ##.##.## kubelet[1354093]: E1121 09:42:18.208475 1354093 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2024-11-17 15:32:57 +0000 UTC
Nov 21 09:42:18 ##.##.## kubelet[1354093]: E1121 09:42:18.208557 1354093 run.go:74] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
Nov 21 09:42:18 ##.##.## systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 09:42:18 ##.##.## systemd[1]: kubelet.service: Failed with result 'exit-code'.

 

Environment

3.0, 3.1, 3.2

Cause

• When the application is booted for the first time after 1 year, kubelet certs are already expired. Its only Kubelet certs which are affected as our automation will take care of control plane certs during first boot. 
• After first boot, Kubelet cannot auto renew as expected as it cannot work with expired cert. Kubelet auto renews when almost 70% expiration date as passed. But in this case, as the certs were created in buildtime. 

 

 

Resolution

The attached update-kubelet-certs_382787.sh script will rotate the kubelet certificate and wait for the node and the TCX installer to install all the resources.

• Download the update-kubelet-certs_382787.sh and move it to the TCA-CP appliance /tmp folder.
• Log into the TCA-CP and change to the /tmp folder.
• Switch to root and apply the read | write | execute permissions to the script:
  su
chmod 755 update-kubelet-certs_382787.sh
• Run the script
  ./update-kubelet-certs_382787.sh