How to increase kubelet log level on vSphere with Tanzu's Supervisor Control Plane VM's

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:
There are 8 levels of kubelet logging with the max being set to level 8 (ie --v=8) 
The default log level will fail to present necessary troubleshooting messages in certain scenarios, increasing the log level can help if in-depth kubelet investigation is required. 

NOTE: Make sure to revert these changes after scoping completion to avoid logs filling up rapidly and overloading the node's disk space / resources. 

Environment

VMware vSphere 8.0 with Tanzu
VMware vSphere 7.0 with Tanzu

Resolution

1. SSH into the Supervisor Control Plane VM: https://knowledge.broadcom.com/external/article?legacyId=90194
2. Backup to the kubeadm-flags.env file: 

# cp /var/lib/kubelet/kubeadm-flags.env /root/kubeadm-flags.env.backup
 

3. Edit the file to include the --v=X where X is the level you want to log. Can be 1 through 8.

# vi /var/lib/kubelet/kubeadm-flags.env

# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --pod-infra-container-image=vmware/pause:1.25.6 --v=8"
 

 

4. Reload daemon and restart service. 

# systemctl daemon-reload

# systemctl restart kubelet

# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Mon 2023-12-04 20:18:03 UTC; 3s ago
     Docs: http://kubernetes.io/docs/
 Main PID: 82437 (kubelet)
...

 

5. Revert changes when done and re-run step 4 to reload/restart kubelet. If kubelet crashes after making the change, there is likely a typo in the file.