How to run TCPDUMP on multiple TAS components simultaneously

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

There are scenarios where we need to run tcpdump on all instances of a specific VM type such as gorouters and Diego cells. This article describes a simple way to simultaneously start and stop tcpdump on these components.

Environment

OS: All Linux Stemcells

Resolution

1. We can start a tcpdump on all instances of a given job using bosh ssh. In the example below we are capturing all traffic out of eth0. When possible, always add a tcpdump filter to narrow the scope of the dump. The below examples uses options "-C 256" and "-W 4" which informs tcpdump to take a rolling trace that creates 4 files with a max size of 256MB each. You need ensure /tmp has at least 1GB of free space if using this example or adjust these args to meet the needs of you sample:
 

Note:  nohup is required to keep the trace running when the shell exits.  But you also need the sleep at the end in order to given nohup enough time to fork the pid from the parent process before the bosh ssh shell exits. 

this example starts a tcpdump on all gorouters and will save up to 1GB of data in /tmp/

bosh ssh router -c '$(nohup sudo tcpdump -i eth0 -C 256 -W 4 -s 256 -w /tmp/`cat /var/vcap/instance/name`-`cat /var/vcap/instance/id`-`date +%s`.trc tcp port 8443 or tcp port 443 &> /tmp/trace.out &); sleep 3'

2. Check if things are running

bosh ssh router -c "sudo ps -ef | egrep tcpdump"

3. Stop the tcpdump on all gorouters

bosh ssh router -c "sudo killall tcpdump"

4. Download trace data into current working directory
Note: this example assumes there are 6 gorouters.  Adjust the for loop to match the number of gorouters.

for i in 0 1 2 3 4 5; do bosh scp router/${i}:/tmp/*.trc* .; done

5. Given that the tcpdump will name the files "<job>-<instance GUID>-<date>.trc", it is always a good practice to capture the output of "bosh vms" for all deployments so you can easily map VM IP addresses.