How to turn up debug levels on Kubernetes Master node components in PKS

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Pivotal Container Service (PKS) deploys and manages the Kubernetes components of a cluster through BOSH. BOSH manages them these clusters through jobs. The BOSH job configuration is put into a bpm.yml configuration file on the master nodes.

In Kubernetes, you adjust the default Kubernetes log level for a component by updating the "--v=X" flag.

On a Kubernetes Master node in PKS, you adjust the default log level for a component by simply updating the bpm.yml file to include the Kubernetes log setting you want. Then restart the Kubernetes component.

The Kubernetes component will then log with the appropriate log level you set until you change it back, or the Kubernetes cluster is upgraded or the Master node is redeployed.

Note: The manual changes should not be expected to remain during these related events. This is by design.

If your Kubernetes cluster is running in Multi-Master (HA) mode that is, if it has 3, 5, or more Master Nodes, make sure to utilize the following article to identify your Master Leader component(s): How to identify the LEADER for Kubernetes Master endpoints in a Multi-Master Cluster configuration in PKS.

Environment

TKGI environment

Resolution

To resolve this issue, follow the below instructions: 

1. Use BOSH to ssh to the Master node in question.

## Use bosh to ssh to the Kubernetes Master node of the bosh service-instance for the Kubernetes cluster
export clusteruuid=$(pks cluster clustername --json | jq -r .uuid)
bosh -d service-instance_${clusteruuid} ssh master/0
sudo -i

If you are logged in to the PKS API, you can target the specific Kubernetes cluster Master with the following command:

## Make sure you are logged in to pks api.  
pks login -a <pks-api-address> -u <user> -k

 

2. Change to the directory of the Kubernetes component you want to turn debugging levels up on. For example: 

cd /var/vcap/jobs/kube-apiserver/config
cd /var/vcap/jobs/kube-controller-manager/config
cd /var/vcap/jobs/kube-scheduler/config

 

3. Make a copy of the existing bpm.yml:

cp bpm.yml bpm.yml_orig

 

4. Edit the bpm.yml file:

vim bpm.yml

 

5. Update the "args:" section to include the "--v=X " log setting you want. For example, setting to log level 9 in /var/vcap/jobs/kube-scheduler/config/bpm.yml, would look like:

---
processes:
- name: kube-scheduler
  executable: /var/vcap/packages/kubernetes/bin/kube-scheduler
  args:

  - --config=/var/vcap/jobs/kube-scheduler/config/config.yml

  - "--v=9"

 

6. Save the file. 

 

7. Restart the BOSH job. In the above example, kube-scheduler is restarted with the following command:

monit restart kube-scheduler

 

8. Confirm the BOSH job goes back into a running state with:

monit summary

Additional Information

If your K8s cluster is running in Multi-Master (HA) mode. That is, if it has 3, 5 or more Master Nodes, make sure to utilize the following article to identify your Master Leader component(s): Reference This KB