How to locate source app which is generating traffic to specified destination on Tanzu Application Service platform

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When an app deployed on TAS platform generates traffic to remote endpoint, for example LDAP service, the remote side would not be able to know which app is sending the request if without any identify info in the request, because source container IP is being hidden once the traffic leaving diego cell, or SNAT is used on the route. 

For example, the security team is going to retire a legacy LDAP server, they want to understand what apps on TAS are still accessing the LDAP server. This KB describes the steps to locate source app which is generating traffic to a remote endpoint.

Environment

Product Version: 2.11

Resolution

1. Run tcpdump to capture the traffic to the specified remote endpoint. If it's not known from which diego cell, tcpdump has to be run on all diego cells. For example, the LDAP service is at IP 10.10.60.21 and port 636. Be aware of capture tcpdump with `-i any` so that container network traffic can also be included. 

$ sudo tcpdump -i any -n host 10.10.60.21 and port 636
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

11:09:01.245479 IP 10.255.66.29.53962 > 10.10.60.21.636: Flags [S], seq 3656171729, win 65472, options [mss 1364,sackOK,TS val 3256598315 ecr 0,nop,wscale 7], length 0
11:09:01.245514 IP 10.10.10.25.53962 > 10.10.60.21.636: Flags [S], seq 3656171729, win 65472, options [mss 1364,sackOK,TS val 3256598315 ecr 0,nop,wscale 7], length 0
...

2. In above example, two TCP SYN packets were captured. Diego cell IP 10.10.10.25 is not needed, we'll use container IP 10.255.66.29 to find the source app. 

3. Query actual LRP with the container IP, in response, the first half (41346d40-d25b-4ff6-aab5-e3ec1194335f) of process_guid is the app GUID. 

$ cfdot actual-lrps | jq -s '.select(.instance_address=="10.255.66.29")'
{
  "process_guid": "41346d40-d25b-4ff6-aab5-e3ec1194335f-ec1abebc-75ac-4cc7-9f21-1c626105f4df",
  "index": 0,
  "domain": "cf-apps",
  "instance_guid": "56ca1541-de5f-4aa5-4a90-6b21",
  "cell_id": "35a4004c-27c5-4406-8f34-9aa5e1fb2dbc",
  "address": "10.10.10.25",
  "ports": [
    {
      "container_port": 8080,
      "host_port": 61016,
      "container_tls_proxy_port": 61001,
      "host_tls_proxy_port": 61018
    },
    {
      "container_port": 2222,
      "host_port": 61017,
      "container_tls_proxy_port": 61002,
      "host_tls_proxy_port": 61019
    }
  ],
  "instance_address": "10.255.66.29",
  "preferred_address": "HOST",
  "crash_count": 0,
  "state": "RUNNING",
  "since": 1637576043088161000,
  "modification_tag": {
    "epoch": "2049c648-4175-4b11-67f7-ea0a051af738",
    "index": 2
  },
  "presence": "ORDINARY"
}