Correlating DNS queries to Applications running on Tanzu Platform for Cloud Foundry

Products

• VMware Tanzu Application Service
• VMware Tanzu Application Service

Issue/Introduction

DNS requests forwarded from applications to external hostnames often appear in upstream DNS server logs with Diego Cell IPs, because those applications use BOSH DNS as their recursor, and BOSH DNS runs directly on the Diego Cells that also runs the apps. It is sometimes necessary to identify which apps are making specific queries to help network troubleshooting. This document outlines ways to identify the applications making a given query on a given diego cell. 

Resolution

The most common way for apps to cause DNS queries from a Diego Cell is by making them directly through processes run in the app container. To determine which app is making a query, start by running a `tcpdump` looking for the specific DNS traffic, on the Diego Cell in question:

# DNS_QUERY=baddomain.example.com
# sudo tcpdump -i any -nn -s0 port 53 | grep -i $DNS_QUERY
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
17:17:11.268894 s-010255013006 In  IP 10.255.13.6.52072 > 169.254.0.2.53: 9010+ [1au] A? baddomain.example.com. (62)
17:17:11.269153 eth0  Out IP 10.0.4.8.43655 > 8.8.8.8.53: 9010+ [1au] A? baddomain.example.com. (62)
17:17:24.960348 s-010255013006 In  IP 10.255.13.6.56908 > 169.254.0.2.53: 516+ A? baddomain.example.com. (39)
17:17:24.960562 eth0  Out IP 10.0.4.8.33996 > 8.8.8.8.53: 516+ A? baddomain.example.com. (39)


Queries listed here coming from an s-##### interface are coming directly from the app container. The container IP is the non-169.254.0.2 address (10.255.13.6 in this case) from the query. To translate that IP into an app GUID, run the following on a Diego Cell:


# IP=10.255.13.6
# cfdot actual-lrps | jq --arg ip "${IP}" --slurp '.[] | select(.instance_address == $ip) | .process_guid | .[0:36]'
"428d193d-39bd-4ad1-97da-d97149946cf3"


However, if in the `tcpdump` output, there is no traffic originating from an s-##### interface, it is possible that the DNS lookups are occurring the result of a syslog drain configured to use the domain in question. To find the app GUID for apps  using this as a drain_url, run the following on a Diego Cell:

DNS_QUERY=baddomain.example.com
cfdot desired-lrps | jq --arg dns_name "${DNS_QUERY}" --slurp '.[] | select(.action.codependent.actions[].run.env[].value | contains($dns_name)) | .process_guid | .[0:36]'
"428d193d-39bd-4ad1-97da-d97149946cf3"

Lastly you get information about the application, and its org/space/processes via:

# cf curl "/v3/apps/${app_guid}"