How to enable debug mode for Consul

Products

• VMware Tanzu Application Service for VMs

Environment

Resolution

• Connect via 'bosh ssh' to the VM (either Consul VM acting as Consul server, or other VMs as Consul agent, like Cloud Controller, Diego Brain, etc).
• 'sudo su -' to root account. 
• Edit the configuration file /var/vcap/jobs/consul_agent/config/config.json. 

The default value is "info" for "log_level". You can change it to "trace", "debug", "warn", or "err". 

This is a sample file from a testing environment. "log_level" is changed to debug now. 

{"server":true,"domain":"cf.internal","datacenter":"dc1","data_dir":"/var/vcap/store/consul_agent","log_level":"debug","node_name":"consul-server-partition-1506967bb2eea78b2b46-0","ports":{"dns":53},"rejoin_after_leave":true,"retry_join":["192.168.27.72"],"bind_addr":"192.168.27.72","disable_remote_exec":true,"disable_update_check":true,"protocol":2,"verify_outgoing":true,"verify_incoming":true,"verify_server_hostname":true,"ca_file":"/var/vcap/jobs/consul_agent/config/certs/ca.crt","key_file":"/var/vcap/jobs/consul_agent/config/certs/server.key","cert_file":"/var/vcap/jobs/consul_agent/config/certs/server.crt","encrypt":"Lfs9s5z/2qAqmjQAoWs+DA==","bootstrap_expect":1}

• Run 'consul reload' command to reload the configuration file. The consul command is always in /var/vcap/jobs/consul_agent/packages/consul/bin/ directory. 

This is the output from a testing environment. 

root@ce241448-d9cd-4106-8a81-b87c86e0c577:/var/vcap/sys/log/consul_agent# /var/vcap/jobs/consul_agent/packages/consul/bin/consul reload
Configuration reload triggered

• You should see log debug entries that look like this: 

root@ce241448-d9cd-4106-8a81-b87c86e0c577:/var/vcap/sys/log/consul_agent# tail -f consul_agent.stdout.log 
 2016/08/02 02:37:30 [INFO] memberlist: Suspect diego-cell-partition-1506967bb2eea78b2b46-0 has failed, no acks received
 2016/08/02 02:37:36 [INFO] memberlist: Suspect diego-cell-partition-1506967bb2eea78b2b46-0 has failed, no acks received
 2016/08/02 05:28:52 [INFO] agent.rpc: Accepted client: 127.0.0.1:53139
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: LAN keyring exists but -encrypt given, using keyring
==> WARNING: WAN keyring exists but -encrypt given, using keyring
==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
 2016/08/02 05:28:52 [DEBUG] agent: Service 'consul' in sync
 2016/08/02 05:29:15 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.75:8301
 2016/08/02 05:29:45 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.80:8301
 2016/08/02 05:30:11 [DEBUG] agent: Service 'consul' in sync
 2016/08/02 05:30:15 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.82:8301
 2016/08/02 05:30:25 [DEBUG] memberlist: TCP connection from: 192.168.27.80:41909
 2016/08/02 05:30:45 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.82:8301
 2016/08/02 05:31:10 [DEBUG] memberlist: TCP connection from: 192.168.27.83:52494
 2016/08/02 05:31:15 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.83:8301
 2016/08/02 05:31:33 [DEBUG] memberlist: TCP connection from: 192.168.27.91:34780
 2016/08/02 05:31:41 [DEBUG] memberlist: TCP connection from: 192.168.27.84:55011
 2016/08/02 05:31:45 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.91:8301
 2016/08/02 05:31:58 [DEBUG] agent: Service 'consul' in sync

Impact

Enabling debug mode will impact the system performance and may also fill up the ephemeral disk, so we strongly recommend to use it only for development or troubleshooting purpose.

 

Additional Information

Related article:

Consul fails to start during upgrade in Cloud Foundry

Consul failing "x/y nodes reported success"