How to collect a packet capture from a container and move it to your local machine.
Article ID: 298167
Updated On:
Products
VMware Tanzu Application Service for VMs
Issue/Introduction
This document provides the steps to collect a packet capture while connected via SSH to the application container using the tcpdump method and move it to your local machine for further analysis.
Pre-requisites:
- Bosh CLI installed. If not installed, this link provides the steps to complete this task.
- CF CLI. If not installed, you can follow the steps from this document.
Environment
Product Version: 4.0
Resolution
- Using cf cli get the app guid you want to trace
-
cf app simple-http --guid 936cd7bb-9216-4f9e-8911-28164104f7ae
-
- ssh to any Diego cell so you have access to the cfdot cli. This will help you find which cells the app is running on. Replace "APP_GUID=<GUID>" with the GUID from the previous step.
-
diego_cell/7913cae8-c9f3-4f0a-a9d6-e0911b395cad:~# APP_GUID=936cd7bb-9216-4f9e-8911-28164104f7ae cfdot actual-lrps | jq -cr --arg app_guid "$APP_GUID" '. | select(.process_guid | startswith($app_guid))| {guid: .instance_guid, address: .instance_address, cell: .cell_id}' {"guid":"99b1d7ed-5edd-4b81-71fa-dc62","address":"10.255.255.2","cell":"83de8232-d9f1-45a8-9994-1179657442e4"} {"guid":"61fb9f91-7d7d-413e-598e-99c2","address":"10.255.255.1","cell":"7913cae8-c9f3-4f0a-a9d6-e0911b395cad"}
-
- ssh to the Diego cell that is running the app container you want to trace and find the a process id that is running in the app. Use the GUID from previous output in the below command. Also make sure the Diego cell you are on matches the above ip address.
-
diego_cell/7913cae8-c9f3-4f0a-a9d6-e0911b395cad:~# sudo /var/vcap/packages/runc/bin/runc --root /run/containerd/runc/garden state 61fb9f91-7d7d-413e-598e-99c2 | /var/vcap/packages/cfdot/bin/jq .pid 3709610
-
- Using the PID from the above output you can start capturing a tcpdump. If you want change "my-app-trace" in the filename to something that resembles the problem you are debugging. Most apps will listen on port 8080 ( unencrypted ).
- Somethings to know about the below tcpdump command arguments
- -i any
- says to listen on any interface.
- -s 0
- when -s is set to zero all bytes will be captured in every packet. You can set this to 256 in cases where you want to capture the headers but drop the body of the packet. This can help keep the trace small for large captures.
- -C 256
- with this value each tcpdump file will not exceed 256mb ( give or take a few ). Paired with -W you can set the max amount of data collected before tcpdump will start truncating existing files.
- -W 4
- This value of 4 means tcpdump will create a maximum of 4 files with the size value of -C, which is 256mb in this example. That means with these args the tcpdump will only capture 1GB of data spread over 4 files number 0,1,2,3. When file 3 reaches 256MB tcpdump will truncate file 0 and continue capturing new data to file 0. This is called a rolling tcpdump. Using -C and -W protects the system from running out of disk space during large captures.
- -i any
-
diego_cell/7913cae8-c9f3-4f0a-a9d6-e0911b395cad:/tmp# sudo nsenter -t 3709610 -n tcpdump -i any -s 0 -C 256 -W 4 -w /tmp/`cat /var/vcap/instance/name`-`cat /var/vcap/instance/id`-my-app-trace.trc tcp port 8080 tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 256 bytes 13 packets captured 32 packets received by filter 0 packets dropped by kernel
- Somethings to know about the below tcpdump command arguments
- If you need to know what ports your app is listening on you can run this command. Modify the tcpdump command to support the port you want to trace.
-
diego_cell/7913cae8-c9f3-4f0a-a9d6-e0911b395cad:/tmp# nsenter -t 3709610 -a netstat -anp | egrep LISTEN tcp 0 0 127.0.0.1:61003 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 25/simple-http tcp 0 0 0.0.0.0:61443 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61443 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61443 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61443 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:2222 0.0.0.0:* LISTEN 7/diego-sshd tcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 79/envoy tcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 79/envoy
-
- the app trace will be saved in the /tmp folder on the diego cell. Note there might be multiple files depending on what values you put for -C and -W args.
-
diego_cell/7913cae8-c9f3-4f0a-a9d6-e0911b395cad:/tmp# ls -l /tmp/diego_cell-7913cae8-c9f3-4f0a-a9d6-e0911b395cad-my-app-trace.trc0 -rw-r--r-- 1 tcpdump tcpdump 2141 May 23 18:31 /tmp/diego_cell-7913cae8-c9f3-4f0a-a9d6-e0911b395cad-my-app-trace.trc0
-
- Now you can SCP the file off of the system
- If the capture is small enough you can move it to a folder in /var/vcap/sys/log. The normal Operations Manager log bundle download will capture all logs in this folder. If you choose this method simply move the files, then go to Operations Manager and trigger a log bundle download for the given diego cell.
- Alternatively you can bosh scp the file to your local machine
-
bosh scp diego_cell/7913cae8-c9f3-4f0a-a9d6-e0911b395cad:/tmp/diego_cell-7913cae8-c9f3-4f0a-a9d6-e0911b395cad-my-app-trace.trc* ~/Documents/
-
Feedback
Yes
No
Powered by
