How to monitor TAS NATS traffic with NATS CLI

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

TAS(Tanzu Application Service) NATS component is being used as message bus within TAS deployment, for example

router-emitter to register/unregister routes on Gorouter
metrics-discovery-registrar to transfer scrape_targets

To troubleshoot app misrouting issue, or missing system component route issue, it's helpful to monitor NATS traffic if a corresponding route is being successfully transferred / registered or unexpectedly unregistered.

Environment

Product Version: 2.11

Resolution

This article describe steps how to monitor TAS NATS traffic with NATS CLI.

1. Download NATS CLI, please check the github release and download any other version or for other platform, this example download v0.0.26 for Linux amd64 platform.

wget https://github.com/nats-io/natscli/releases/download/0.0.26/nats-0.0.26-linux-amd64.zip

2. Unzip above zip file, put the `nats` binary at any jumpbox which can access TAS NATS VM at port 4222. Add the binary directory to $PATH and set it as executable(chmod +x nats).

3. Find the NATS credential at either location

on Gorouter VM: /var/vcap/jobs/gorouter/config/gorouter.yml
on NATS VM: /var/vcap/jobs/nats/config/nats.conf

4. Subscribe for receiving route registrar/unregistrar messages

$ nats --server="NATS_VM_IP:4222" --user=nats --password="*************"  sub "router.>"
21:58:30 Subscribing on >

[#1] Received on "router.register"
{"uris":["api.system.<DOMAIN>/networking"],"host":"10.213.60.14","tls_port":4002,"tags":{"component":"NetworkPolicyServer"},"private_instance_id":"01c9819f-c43b-4ef6-763f-db10d98fa373","server_cert_domain_san":"network-policy-server.service.cf.internal"}

[#2] Received on "router.register"
{"uris":["scs-mirror-service.system.<DOMAIN>"],"host":"10.213.60.40","port":50505,"tags":{"component":"scs-mirror-service"},"private_instance_id":"e3294321-6b4c-4fd9-7812-4687ffecfa89"}

[#3] Received on "router.register"
{"host":"10.213.60.23","port":61016,"tls_port":61021,"uris":["myapp.apps.<DOMAIN>"],"app":"74066def-6bdc-4f14-bde8-b47a08f297dc","private_instance_id":"34f5baef-3eae-43d0-4e34-9fb9","private_instance_index":"0","server_cert_domain_san":"34f5baef-3eae-43d0-4e34-9fb9","tags":{"app_id":"74066def-6bdc-4f14-bde8-b47a08f297dc","app_name":"myapp","component":"route-emitter","instance_id":"0","organization_id":"abf39819-0c7c-4985-b53c-41c59f0c4947","organization_name":"system","process_id":"74066def-6bdc-4f14-bde8-b47a08f297dc","process_instance_id":"34f5baef-3eae-43d0-4e34-9fb9","process_type":"web","source_id":"74066def-6bdc-4f14-bde8-b47a08f297dc","space_id":"0938cc81-66fb-478d-8a31-0fb541a39f27","space_name":"my-space"}}
...

5. If you would like to capture all traffic over NATS other than router messages only, please change the subscription string to '>'.

$ nats --server="NATS_VM_IP:4222" --user=nats --password="*************"  sub ">"
21:58:30 Subscribing on >

If it's not allowed to use NATS CLI in the environment, please refer another KB https://knowledge.broadcom.com/external/article/297651/how-to-monitor-nats-traffic-on-the-gorou.html for capture NATS traffic with tcpdump.