How to monitor NATS traffic on the Gorouter

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

You need to confirm that the Gorouter is receiving announcements from a Diego Cell's rep process. To do this you can packet capture the NATS traffic on a Gorouter VM to confirm the announcements are being received. One might do this to debug routes not found by a Gorouter for incoming app URLs, or to confirm a Diego Cell's rep process is sending the <IP Address:Port> of one of the Cell's hosted apps.

Environment

Resolution

1. Log in to the Operations Manager console and SSH to a Router VM.

bosh deployments
bosh -d cf-dd5556db39fdde2c5c84 vms
bosh -d cf-dd5556db39fdde2c5c84 ssh router/04751932-18ba-4242-ba1a-f6327fd24712

2. Listen for the NATS announcements for an app with a hostname spring-music. Give it a minute or two to show up.  

sudo -i tcpdump -A -nn -i eth0 'port 4222' | grep spring-music

3. Network packets can be fragmented, so if you do not get readable data it would be better to save the tcpdump to a file

tcpdump -i eth0 'port 4222' -w /var/tmp/nats.tcpdump

4. Cancel it after a minute or two and read the file back with

tcpdump -A -nn  -r /var/tmp/nats.tcpdump 'port 4222' | grep spring-music

5. Otherwise, download the file to view in a network capture utility like Wireshark.

chmod +r /var/tmp/nats.tcpdump

6. Then back on the Operations Manager, copy the file from the router.

bosh -d cf-dd5556db39fdde2c5c84 scp router/04751932-18ba-4242-ba1a-f6327fd24712:/var/tmp/nats.tcpdump nats.tcpdump

7. This is an example NATS message advertising the spring-music app...

{"host":"10.193.84.34","port":61028,"tls_port":61030,"uris":["spring-music-unashamed-range.cfapps-19.###-##.###.######.io"],"app":"fdb25a52-a203-415a-a3fd-3bd0b9180c7f","private_instance_id":"03c12d80-30f5-48a1-5a17-477e","private_instance_index":"0","server_cert_domain_san":"03c12d80-30f5-p48a1-5a17-477e","tags":{"component":"route-emitter"}}

8. From the above step, you see the app is in the Diego cell with the address 10.193.84.34:61028. This confirms that the Diego cell with IP 10.193.84.34 is sending app routes.

If you have multiple NATS servers, it may be helpful to view the messages for each server one at a time. You can get the IP addresses from the Operations Manager /PAS/Status page, or from the routers config file /var/vcap/jobs/gorouter/config/gorouter.yml.

tcpdump -A -nn  -r /var/tmp/nats.tcpdump 'port 4222 and src 10.193.84.19' | grep spring-music
tcpdump -A -nn  -r /var/tmp/nats.tcpdump 'port 4222 and src 10.193.84.55' | grep spring-music