How to enable debug logging level for the pods in knative-serving namespace

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Sometimes it's necessary to retrieve the debug logs for pods in knative-serving namespace for troubleshooting purpose. This KB article is to show you the method of enabling and disabling the debug logging level for the pods in knative-serving namespace.

Environment

Product Version: 1.6

Resolution

1 - When we look into the config-logging configmap, we can see the default logging level for the pods in knative-serving namespace is "info".

$ kubectl get configmap/config-logging -n knative-serving -o jsonpath={.data."_example"} | grep "loglevel"
loglevel.controller: "info"
loglevel.autoscaler: "info"
loglevel.queueproxy: "info"
loglevel.webhook: "info"
loglevel.activator: "info"
loglevel.hpaautoscaler: "info"
loglevel.net-certmanager-controller: "info"
loglevel.net-istio-controller: "info"
loglevel.net-contour-controller: "info"

2 - We can use kubectl patch command to enable the "dubug" logging level for the pods in knative-serving namespace. Let's take loglevel.net-contour-controller as a sample:

$ kubectl patch configmap/config-logging \
    --namespace knative-serving \
    --type merge \
    --patch '{"data":{"loglevel.net-contour-controller":"debug"}}'

3 - Above logging level change is picked up immediately and no recreation of the pods is required. The logging level change can also be confirmed from the deployment/pod logs.

$ kubectl -n knative-serving logs deploy/net-contour-controller
......
{"severity":"INFO","timestamp":"2023-11-22T03:55:45.434590308Z","logger":"net-contour-controller","caller":"logging/config.go:225","message":"Updating logging level for net-contour-controller from info to debug.","commit":"732f746"}

4 - We can use kubectl patch command again to revert the logging level back to "info".

$ kubectl patch configmap/config-logging \
    --namespace knative-serving \
    --type json \
    --patch '[{"op": "remove", "path": "/data/loglevel.net-contour-controller"}]'