How to enable debug logging for NATS v2

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This Knowledge Base (KB) article details steps on how to enable debug logging for the nats v2 service running on the nats instance group in Tanzu Application Service for VMs (TAS).

Starting in nats release v46 the underlying package nats-server has been updated from v1 to v2. This change has been introduced in the following TAS versions:

• TAS v2.11.26+
• TAS v2.13.14+
• TAS v3.0+
• TAS v4.0+
• TAS v5.0+

For more information on how to confirm your nats service is running on v2 please see this KB How to confirm your nats servers are running NATs v2.

To obtain the exact nats-server version running on the nats VM, ssh on a nats VM and run nats-server --version as seen below:

nats/f1d01b98-918d-46c3-8388-e969e39479dc:/var/vcap/jobs/nats# /var/vcap/packages/nats-server/bin/nats-server --version

nats-server: v2.8.4

In the event you need to enable debug logging for a nats server, the following method may be used. Please note the below method works for both tls and non-tls nats.

Environment

Product Version: 4.0

Resolution

STEP 1
SSH on the nats VM


STEP 2
Become sudo and cd to the nats config directory:

sudo su -
cd /var/vcap/jobs/nats/config

STEP 3
Make a copy of the original config file for safe keeping:

cp nats.conf /tmp

STEP 4
Edit the nats.conf file to include the logging settings similar to how it is defined in the nats documentation. The config file should contain something similar to:

debug: true
trace: false
logtime: true
log_file: "/tmp/nats-server.log"

Even though log_file attribute specifies "/tmp/nats-server.log" the log file will not be created in the VM /tmp directory but instead at /var/vcap/data/nats/tmp/


STEP 5
Validate how much ephemeral disk space is available at path /var/vcap/data:

nats/f1d01b98-918d-46c3-8388-e969e39479dc:/var/vcap/jobs/nats/config# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            97M  9.4M   88M  10% /run
/dev/sda1       4.9G  2.7G  2.0G  59% /
tmpfs           485M     0  485M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sdb2       6.9G  238M  6.3G   4% /var/vcap/data
tmpfs            16M  320K   16M   2% /var/vcap/data/sys/run
tmpfs            97M     0   97M   0% /run/user/1001

In this example, it shows we have 6.3GB available on the ephemeral disk. We will set up a logrotate file to allow up to 500MB of this disk space for the nats debug logs in step 6.


STEP 6
Setup logrotate so that the debug logs will not exceed the ephemeral disk space on the VM, in this example we will limit it to 500MB. Create a file named nats at directory /etc/logrotate.d/ with the following contents:

/var/vcap/data/nats/tmp/nats-server.log {
    copytruncate
    rotate 5
    compress
    missingok
    size 100M
}

STEP 7
Restart nats-wrapper process:

monit restart nats-wrapper

STEP 8
Ensure the log files are populating with data:

nats/f1d01b98-918d-46c3-8388-e969e39479dc:/etc/logrotate.d# ls -lh /var/vcap/data/nats/tmp

-rw-r----- 1 vcap vcap  19K Oct  9 23:24 nats-server.log

Additional Information

When the vm that has the changes implemented will be recreated, it will go back to default settings.