How to turn up debug levels on Kubernetes Worker node components in PKS

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Pivotal Container Service (PKS) deploys and manages the Kubernetes components of a cluster through BOSH. BOSH manages them these clusters through jobs. 

In Kubernetes, you adjust the default Kubernetes log level for a component by updating the "--v=X" flag.

On a Kubernetes Worker node in PKS, you adjust the default log level for a
component by simply updating the *_ctl file of the Kubernetes component with the Kubernetes log setting you want. Then restart the Kubernetes component.

The Kubernetes component will then log with the appropriate log level you set until you change it back, or the Kubernetes cluster is upgraded or the Master node is redeployed.

Note: The manual changes should not be expected to remain during these related events. This is by design.

Environment

Resolution

To resolve this issue, follow the below instructions: 

1. Use BOSH to ssh to the Worker node you are troubleshooting.

If you are logged in to the PKS API, you can target the specific kubernetes cluster Worker with the following command:

## If you are logged in to pks api.  Example:
pks login -a <pks-api-address> -u <user> -k

2. Then use BOSH to ssh to the Kubernetes Worker node of the service-instance for the Kubernetes cluster:

export clusteruuid=$(pks cluster clustername --json | jq -r .uuid)
bosh -d service-instance_${clusteruuid} ssh worker/0
sudo -i

3. Follow the below steps to change the log level of the Kubernetes worker component, kubelet.

• Change to the directory of the kubelet.

cd /var/vcap/jobs/kubelet/bin

• Verify that you return the following single exact line before continuing:

sed -n '/kubelet \\/p' kubelet_ctl
##Return:  kubelet \

• If successful, then edit the *_ctl file and make a copy of this file.  For example: 

sed -i.backup 's/kubelet \\/kubelet --v=9 \\/' kubelet_ctl

• Restart the kubelet to make the log level change take effect. 

monit restart kubelet

• To revert back to default log level of kubelet, run the following command. 

cp /var/vcap/jobs/kubelet/bin/kubelet_ctl.backup /var/vcap/jobs/kubelet/bin/kubelet_ctl
monit restart kubelet

4. Follow the below steps to change the log level of the Kubernetes worker component, kube-proxy.

• Change to the directory of the kube-proxy;

cd /var/vcap/jobs/kube-proxy/bin

• Verify that you return the following single exact line before continuing:

sed -n '/kube-proxy \\/p' kube_proxy_ctl
##Return: kube-proxy \

• If successful, then edit the *_ctl file and make a copy of this file. For example: 

##Setting log level 9 for V logs:
sed -i.backup 's/kube-proxy \\/kube-proxy --v=9 \\/' kube_proxy_ctl

• Restart the kube-proxy to make the log level change take effect. 

monit restart kube-proxy 

• To revert back to default log level of kube-proxy, run the following command:

cp /var/vcap/jobs/kube-proxy/bin/kube_proxy_ctl.backup /var/vcap/jobs/kube-proxy/bin/kube_proxy_ctl
monit restart kube-proxy

Additional Information

https://community.pivotal.io/s/article/how-to-turn-up-debug-levels-on-kubernetes-master-node-components-in-pks