How to collect a packet capture from a container and move it to your local machine.
Article ID: 298167
Updated On:
Products
VMware Tanzu Application Service for VMs
Issue/Introduction
This document provides the steps to collect a packet capture while connected via SSH to the application container using the tcpdump method and move it to your local machine for further analysis.
Pre-requisites:
- Bosh CLI installed. If not installed, this link provides the steps to complete this task.
- CF CLI. If not installed, you can follow the steps from this document.
Environment
Product Version: 4.0 - 10.0
Resolution
- Using
cf cliget the app guid you want to trace:# cf app simple-http --guid936cd7bb-9216-4f9e-8911-28164104f7ae - ssh to any Diego cell so you have access to the cfdot cli. This will help you find which cells the app is running on.
- Use the
bosh sshcommand with the cf-<FOUNDATION_ID> and the Diego Cell ID:# bosh vms# bosh -d cf<FOUNDATION_ID> ssh diego_cell/<ID>
Example:bosh -d cf-8a##############af9 ssh diego_cell/7913cae8-c9f3-4f0a-a9d6-e0911b395cad
- Use the
- Replace "APP_GUID=<GUID>" with the GUID from the first step:
# APP_GUID=936cd7bb-9216-4f9e-8911-28164104f7ae
# cfdot actual-lrps | jq -cr --arg app_guid "$APP_GUID" '. | select(.process_guid | startswith($app_guid))| {guid: .instance_guid, address: .instance_address, cell: .cell_id}'{"guid":"99b1d7ed-5edd-4b81-71fa-dc62","address":"10.255.255.2","cell":"83de8232-d9f1-45a8-9994-1179657442e4"}{"guid":"61fb9f91-7d7d-413e-598e-99c2","address":"10.255.255.1","cell":"7913cae8-c9f3-4f0a-a9d6-e0911b395cad"} - ssh to the Diego cell that is running the app container you want to trace and find the process ID that is running in the app. Use the GUID from previous output in the below command. Also make sure the Diego cell you are on matches the above IP address.
# sudo /var/vcap/packages/runc/bin/runc --root /run/containerd/runc/garden state 61fb9f91-7d7d-413e-598e-99c2 | /var/vcap/packages/cfdot/bin/jq .pid3709610 - Using the PID from the above output you can start capturing a tcpdump. If desired, change "
my-app-trace" in the filename to something that resembles the problem you are debugging. Most apps will listen on port 8080 (unencrypted):- Some things to know about the below tcpdump command arguments:
-i any- Says to listen on any interface.
-s 0- when -s is set to zero all bytes will be captured in every packet. You can set this to 256 in cases where you want to capture the headers but drop the body of the packet. This can help keep the trace small for large captures.
-C 256- With this value each tcpdump file will not exceed 256MB (give or take a few). Paired with
-Wyou can set the max amount of data collected before tcpdump will start truncating existing files.
- With this value each tcpdump file will not exceed 256MB (give or take a few). Paired with
-W 4- This value of 4 means tcpdump will create a maximum of 4 files with the size value of
-C, which is 256MB in this example. With these command arguments, the tcpdump will only capture 1GB of data spread over 4 files numbered 0,1,2,3. When file 3 reaches 256MB, tcpdump will truncate file 0 and continue capturing new data to file 0. This is called a rolling tcpdump. Using -C and -W protects the system from running out of disk space during large captures.
- This value of 4 means tcpdump will create a maximum of 4 files with the size value of
-
# sudo nsenter -t 3709610 -n tcpdump -i any -s 0 -C 256 -W 4 -w /tmp/`cat /var/vcap/instance/name`-`cat /var/vcap/instance/id`-my-app-trace.trc tcp port 8080tcpdump: data link type LINUX_SLL2tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 256 bytes13 packets captured32 packets received by filter0 packets dropped by kernel
- Some things to know about the below tcpdump command arguments:
- If you need to know what ports your app is listening on you can run this command. Modify the tcpdump command to support the port you want to trace.
# nsenter -t 3709610 -a netstat -anp | egrep LISTENtcp 0 0 127.0.0.1:61003 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 25/simple-httptcp 0 0 0.0.0.0:61443 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61443 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61443 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61443 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:2222 0.0.0.0:* LISTEN 7/diego-sshdtcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 79/envoytcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 79/envoy - The app trace will be saved in the /tmp folder on the diego cell. Note there might be multiple files depending on what values you put for -C and -W args.
# ls -l /tmp/diego_cell-7913cae8-c9f3-4f0a-a9d6-e0911b395cad-my-app-trace.trc0-rw-r--r-- 1 tcpdump tcpdump 2141 May 23 18:31 /tmp/diego_cell-7913cae8-c9f3-4f0a-a9d6-e0911b395cad-my-app-trace.trc0 - Now you can SCP the file off of the system
- If the capture is small enough you can move it to a folder in
/var/vcap/sys/log. The normal Operations Manager log bundle download will capture all logs in this folder. If you choose this method simply move the files, then go to Operations Manager and trigger a log bundle download for the given diego cell. - Alternatively you can
bosh scpthe file to your local machine:# bosh scp diego_cell/7913cae8-c9f3-4f0a-a9d6-e0911b395cad:/tmp/diego_cell-7913cae8-c9f3-4f0a-a9d6-e0911b395cad-my-app-trace.trc* ~/Documents/
- If the capture is small enough you can move it to a folder in
Feedback
Yes
No
Powered by
