How to change logging level of envoy proxy in TAS application instance

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Envoy proxy is a sidecar process running in the TAS application instance (AI) for TLS communication with the router. The default logging level of envoy proxy is set to Critical.

vcap@########-####-####-####-####:~$ ps -ef|grep envoy
root          91      54  0 May27 ?        00:01:01 /etc/cf-assets/envoy/envoy -c /etc/cf-assets/envoy_config/envoy.yaml --drain-time-s 900 --log-level critical

Sometimes there might be the need to increasing the logging level to produce more logs for troubleshooting purposes. This article describes how to check the logging level of the envoy proxy on TAS AI as well as how to change it. 

 

Environment

Product Version: 2.11

Resolution

As described in the doc, the /logging endpoint could be utilized to check & set the logging level of envoy proxy loggers.

1. cf ssh into an application instance and curl the /logging endpoint to view all loggers' levels:

vcap@########-####-####-####-####:~$ curl -X POST http://localhost:61003/logging
active loggers:
  admin: critical
  aws: critical
  assert: critical
  backtrace: critical
  cache_filter: critical
  client: critical
  config: critical
  connection: critical
  conn_handler: critical
  decompression: critical
  dubbo: critical
  file: critical
  filter: critical
  forward_proxy: critical
  grpc: critical
  hc: critical
  health_checker: critical
  http: critical
  http2: critical
  hystrix: critical
  init: critical
  io: critical
  jwt: critical
  kafka: critical
  lua: critical
  main: critical
  misc: critical
  mongo: critical
  quic: critical
  quic_stream: critical
  pool: critical
  rbac: critical
  redis: critical
  router: critical
  runtime: critical
  stats: critical
  secret: critical
  tap: critical
  testing: critical
  thrift: critical
  tracing: critical
  upstream: critical
  udp: critical
  wasm: critical

2. Set all loggers' level to info:

vcap@########-####-####-####-####:~$ curl -X POST http://localhost:61003/logging?level=info
active loggers:
  admin: info
  aws: info
  assert: info
  backtrace: info
  cache_filter: info
  client: info
  config: info
  connection: info
  conn_handler: info
  decompression: info
  dubbo: info
  file: info
  filter: info
  forward_proxy: info
  grpc: info
  hc: info
  health_checker: info
  http: info
  http2: info
  hystrix: info
  init: info
  io: info
  jwt: info
  kafka: info
  lua: info
  main: info
  misc: info
  mongo: info
  quic: info
  quic_stream: info
  pool: info
  rbac: info
  redis: info
  router: info
  runtime: info
  stats: info
  secret: info
  tap: info
  testing: info
  thrift: info
  tracing: info
  upstream: info
  udp: info
  wasm: info

3. Set some particular logger's level to info:

vcap@########-####-####-####-####:~$ curl -X POST http://localhost:61003/logging?main=info
active loggers:
  admin: critical
  aws: critical
  assert: critical
  backtrace: critical
  cache_filter: critical
  client: critical
  config: critical
  connection: critical
  conn_handler: critical
  decompression: critical
  dubbo: critical
  file: critical
  filter: critical
  forward_proxy: critical
  grpc: critical
  hc: critical
  health_checker: critical
  http: critical
  http2: critical
  hystrix: critical
  init: critical
  io: critical
  jwt: critical
  kafka: critical
  lua: critical
  main: info
  misc: critical
  mongo: critical
  quic: critical
  quic_stream: critical
  pool: critical
  rbac: critical
  redis: critical
  router: critical
  runtime: critical
  stats: critical
  secret: critical
  tap: critical
  testing: critical
  thrift: critical
  tracing: critical
  upstream: critical
  udp: critical
  wasm: critical

Note:
 

• Increasing envoy logging levels might add some performance overhead to the container, which should be used for troubleshooting purposes only. Reset it back to critical once troubleshooting is done. 
• The logging level will be set back to default (critical) after the application instance is restarted.
• For applications with multiple instances, run cf ssh -i to get into each instance to change the logging level respectively

The available log levels are:
 

• trace

• debug

• info

• warning/warn

• error

• critical

• off

Since envoy will stream logs to /dev/stderr by default, run cf logs <app name> to tail the real time logs or cf logs <app name> --recent to retrieve recent logs. Envoy log messages will be shown with component PROXY and level ERR as illustrated in the following example:

2021-05-28T02:43:35.14+0000 [PROXY/0] ERR [2021-05-28 02:43:35.147][91][trace][http] [source/common/http/conn_manager_impl.cc:1569] [C11][S4040853541271744932] encode headers called: filter=0x558b77291130 status=0
2021-05-28T02:43:35.14+0000 [PROXY/0] ERR [2021-05-28 02:43:35.147][91][debug][http] [source/common/http/conn_manager_impl.cc:1710] [C11][S4040853541271744932] encoding headers via codec (end_stream=false):