Collecting Packet Captures for iSCSI Performance Troubleshooting

Products

• VMware vSphere ESXi

Issue/Introduction

Details on how to collect packet captures for iSCSI performance troubleshooting.

Resolution

Collecting the below data is crucial for understanding iSCSI performance:

1. The 3-way TCP handshake for the iSCSI connection. This allows the initial round trip time (iRTT) to be calculated and to capture TCP window scaling information only transmitted during the initial handshake. This is achieved on a test ESXi setup by removing iSCSI sessions, and after beginning the capture, adding them back.  

2. Full packets. 

On a Test ESXi

On an ESXi host which will run only test workloads (no production workloads): 

1. Remove iSCSI sessions

- Identify the iSCSI adapter

# esxcli iscsi adapter list

- Remove the sessions: 
e.g. suppose iSCSI adapter is vmhba68

# esxcli iscsi session remove -A vmhba68

- Confirm no iSCSI sessions remain:

# esxcli iscsi session list -A vmhba68

2. Initiate packet capture:

On an SSH session to the ESXi host, run capture on all vmkernel ports which carry iSCSI traffic:

# tcpdump-uw -i vmk1 -s 0 -w /vmfs/volumes/datastore2/perfData/vmk1.pcap 

where: 
- vmk1 is the vmkernel port carrying iSCSI traffic
- datastore2 is a datastore with sufficient free space and perfData is a directory already created within the datastore

- Incoming and outgoing packets are captured. Full packets are captured (-s 0)

Note:
- If more than one vmkernel port carries iSCSI traffic, a tcpdump-uw should be run for each vmkernel port in parallel via separate SSH sessions to the ESXi host. 

Collect esxtop batch data in parallel, e.g.: 
# esxtop -b -a -d 5 -n 120 > /vmfs/volumes/datastore2/perfData/esxtopOutput.csv 

3. Add iscsi sessions:

# esxcli iscsi session add -A vmhba68

Check that sessions are created:
# esxcli iscsi session list -A vmhba68

4. Initiate test IO 

5. At test completion, stop tcpdump-uw (CTRL + C)

6. Confirm that data has been collected:

tcpdump-uw -r /vmfs/volumes/datastore2/perfData/vmk1.pcap | head -n 10 | less

7. Collect a vm-support log bundle for the host

On a Production ESXi host

If packet captures are being taken on ESXi hosts running production or other workloads, do not remove iSCSI sessions.

1. Initiate packet capture, together with esxtop data (as per step 2 above).

2. If test I/O is being captured, run test I/O.

3. When test is complete, or a sufficient sample of I/O is collected, stop tcpdump-uw (CTRL + C)

4. Confirm that the data has been collected (as per 6 above)

5. Collect a vm-support bundle for the ESXi host.

Additional Information

Further tcpdump-uw options:

- Where the test traffic generates a large number of packets, the number of packets to be collected can be limited using the -c option, e.g.:

# tcpdump-uw -i vmk1 -s 0 -w /vmfs/volumes/datastore2/perfData/vmk1.pcap -c 1000000

- Where the issue may take some time to reproduce, capture a specified number of rotating pcap file of fixed size, e.g.:

# tcpdump-uw -i vmk1 -s 0 -W 10 -C 50 -w /vmfs/volumes/datastore2/perfData/vmk1.pcap

The above command will generate 10 rotating pcap files each 50 MB in size approximately. Stop capture once the issue reproduces and sufficient coverage of the issue has been attained.

- Where the issue may take time to reproduce such that an SSH session may time out or need to be closed, nohup can be utilized to ensure data collection beyond the end of the SSH session. e.g.

nohup tcpdump-uw -i vmk1 -s 0 -W 10 -C 50 -w /vmfs/volumes/datastore2/perfData/vmk1.pcap

or with collection for 2 iSCSI vmkernel ports:

nohup tcpdump-uw -i vmk1 -s 0 -W 10 -C 50 -w /vmfs/volumes/datastore2/perfData/vmk1.pcap & nohup tcpdump-uw -i vmk2 -s 0 -W 10 -C 50 -w /vmfs/volumes/datastore2/perfData/vmk2.pcap &

To check running nohup jobs:

 jobs -l | grep nohup
[2]+  582926 Running             nohup tcpdump-uw -i vmk1 -s 0 -W 10 -C 50 -w /vmfs/volumes/datastore2/perfData/vmk1.pcap
[1]-  582925 Running             nohup tcpdump-uw --i vmk2 -s 0 -W 10 -C 50 -w /vmfs/volumes/datastore2/perfData/vmk2.pcap

To stop the jobs, kill each PID (582926 and 582926 in above output):
kill -9 <nohup pid>

Impact/Risks:
None