Error: failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Expired Kubelet Certs
• Kubelet is failing to start
• In TCA-CP, the command openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate returns notAfter=Nov 17 15:32:57 2024 GMT
• journalctl displays the following errors:

Nov 21 09:42:18 ##.##.## kubelet[1354093]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Nov 21 09:42:18 ##.##.## kubelet[1354093]: I1121 09:42:18.205989 1354093 server.go:412] "Kubelet version" kubeletVersion="v1.26.5+vmware.2"
Nov 21 09:42:18 ##.##.## kubelet[1354093]: I1121 09:42:18.206053 1354093 server.go:414] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 21 09:42:18 ##.##.## kubelet[1354093]: I1121 09:42:18.206557 1354093 server.go:836] "Client rotation is on, will bootstrap in background"
Nov 21 09:42:18 ##.##.## kubelet[1354093]: E1121 09:42:18.208475 1354093 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2024-11-17 15:32:57 +0000 UTC
Nov 21 09:42:18 ##.##.## kubelet[1354093]: E1121 09:42:18.208557 1354093 run.go:74] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
Nov 21 09:42:18 ##.##.## systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 09:42:18 ##.##.## systemd[1]: kubelet.service: Failed with result 'exit-code'.

 

Environment

3.0, 3.1, 3.2

Cause

• When the application is booted for the first time after 1 year, kubelet certs are already expired. Its only Kubelet certs which are affected as our automation will take care of control plane certs during first boot. 
• After first boot, Kubelet cannot auto renew as expected as it cannot work with expired cert. Kubelet auto renews when almost 70% expiration date as passed. But in this case, as the certs were created in buildtime. 

 

 

Resolution

The attached update-kubelet-certs_382787.sh script will rotate the kubelet certificate and wait for the node and the TCX installer to install all the resources.

Run the steps on both the TCA Manager and TCA Control Plane Nodes:

1. Download the update-kubelet-certs_382787.sh and move it to the  appliance /tmp folder.
2. ssh to the node and change to the /tmp folder.
3. Switch to root and apply the read | write | execute permissions to the script:
   su
chmod 755 update-kubelet-certs_382787.sh
4. Run the script
   ./update-kubelet-certs_382787.sh