Packet capture on ESXi using the pktcap-uw tool in ESXi 5.5 and later

Products

VMware vSphere ESXi

Issue/Introduction

The pktcap-uw tool is an enhanced packet capture and analysis tool that can be used in place of the legacy tcpdump-uw tool.

The pktcap-uw tool is included by default in ESXi 5.5 and later versions. This article provides information on using the pktcap-uw tool for packet capture and analysis.

Note: The tcpdump-uw tool can only capture packets/frames at the vmkernel interface level and cannot capture frames at the uplinks, or vSwitch, or virtual port levels. The new pktcap-uw tool allows traffic to be captured at all points within the hypervisor for greater flexibility and improved troubleshooting.

Environment

VMware vSphere ESXi

Resolution

OVERVIEW

As a starting point, bookmark, and visit ESXi Network Troubleshooting Tools

At the above link, scroll down to the section title "Pktcap-uw".

Under that title, look for the following graphic:

Packet captures are done by entering commands at the command line of a session via SSH (or Server Console access via KVM -- “Keyboard, Video (monitor), Mouse.”) with root access privileges.

Capture points

There are a variety of points at which a traffic can be captured, depending on the type of switch in use for example standard vSwitch or a Distributed Virtual Switch (DVS).

And, capture output can be directed to the user's screen and displayed (by parsing the output with the tcpdump-uw tool), or it can be directed to a disk file which can then be downloaded and analyzed with a tool such as Wireshark.

Plan on directing the output to a disk file, please NEVER use the /tmp directory as a place to put the data.

Instead, first determine what datastores are available to the ESXi host using the command:

df -h

Then , change to the datastore

cd /vmfs/volumes/FULL_PATH_TO_DATASTORE

Enter the full path for the FULL_PATH_TO_DATASTORE above.

NOTE: When sending the data to a VSAN datastore, enclose the command in quotes, such as:

cd "/vmfs/volumes/FULL_PATH_TO_DATASTORE"

ALSO NOTE: As a general rule, when doing captures from multiple ESXi hosts, it makes the most sense to use a datastore that is accessible from all of those hosts.

It is also a good idea to create a sub-folder or sub-directory at that datastore, such as, for example, "CASE_12345678" where instead of "12345678", use the Broadcom Support Case Number.

Example:

cd "/vmfs/volumes/FULL_PATH_TO_DATASTORE"

mkdir Case_12345678

Standard vSwitch Capture Points

in vSwitch i.e. switchport and uplink.

Switchport connects to all the vNIC and vmk adapters and uplink is the actual physical NIC (vmnic).

We will first identify the capture point based on the issue and apply the command syntax accordingly.

In the below diagram not the architecture and how VM’s, kernel and physical adapters are connected.

Identifying the active uplink carrying a specific vm/vmk traffic.

Run the command "net-stats -l" and this will list the switchports that are mapped to vm/vmk.

Option 1 to identify the uplink: Run the command "esxtop" and press "n". This will show the mapping between switchport and uplink.

Option 2 to identify the uplink: Run the command "vsish" and press enter. "cat/portsets/vSwitch_name/ports/switchport_number/teamuplink"

Note: For vSwitch name and switchport number refer to "net-stats -l" output.

Option 3 to identify the uplink: Run the command "netdbg vswitch instance list" and press enter.

Advanced Usage: trace multiple ports at the same time

As an example, trace a particular vSwitch port and its associated uplink at the same time:

To get the vSwitch port number, run this command:

net-stats -l
Identify and make a note these parameters:

Port ID returned by the esxtop command — --switchport 12345678

vmnic2 physical port to trace — --uplink vmnic2 -

location of the output pcap file — /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/
Run the pktcap-uw command to capture packets at both points simultaneously:

pktcap-uw --switchport 67108879 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.pcapng & pktcap-uw --uplink vmnic2 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.uplink.vmnic2.pcapng &

Note: The command shell we assume here is the Bash shell. Therefore, if more than one uplink on the vswitch, just add the command after the ending & and add a & at the end.

To stop pktcap-uw tracing with the kill command:

kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)
Run this command to check that all pktcap-uw traces are stopped:

lsof |grep pktcap-uw |awk '{print $1}'| sort -u

When doing only a single capture, omit the "&"

To stop a single capture, press and hold the "CTRL" key and touch "C".

Expect to see a number of messages on the screen as the packets are being captured.

For better viewing while this is happening, start a duplicate session using SSH.

Distributed Virtual Switch (DVS) Capture Points

Per the diagram, there are found different capture points.

1) The closest capture point to the physical infrastructure (i.e. the entry / exit point between the ESXi hypervisor and the physical infrastructure, is called the --uplink capture point.

Example for directing the output to a file:

pktcap-uw --uplink vmnic2 --capture UplinkSndKernel,UplinkRcvKernel -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.uplink.vmnic2.UplinkSndRcvKernel.pcapng &

Example for directing the output to the screen:

pktcap-uw --uplink vmnic2 --capture UplinkSndKernel,UplinkRcvKernel -o - | tcpdump-uw -r - -enn

NOTES:

a) To terminate the capture if when directing the output to a file:

kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)

b) To terminate the capture when directing the output to the screen: Press and hold the CTRL key and touch "C".

2) The capture point closest to the actual virtual machine's virtual NIC is called the --switchport capture point.

Example for directing the output to a file:

pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.VnicTxRx.pcapng &

Example for directing the output to the screen:

pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -o - | tcpdump-uw -r - -enn

3) The other two capture points are shown in the OVERVIEW diagram at the top of this Resiolution section, and generally are only used when working with a Broadcom Technical Support Engineer (TSE), during an investigation via a Broadcom Support Case.

Additional Information

Please note that packet capture data is not stored in ESXi logs.

If additional assistance is needed with troubleshooting, we recommend reaching out to Broadcom support by creating a support case using the instructions at Creating and managing Broadcom support cases

TIPS TO HELP KEEP PACKET CAPTURE FILES SMALLER, TO MAKE ANALYSIS AND COLLABORATION EASIER AND FASTER:

1) If no payload is required to be captured, add the following, for example, a packet size limiting switch, as per this example:

pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -s 256 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.VnicTxRx.pcapng &

Above, the addition of the -s 256 switch will limit the size of each packet captured to the first 256 bytes, which contains the header, which is usually the most important info when troubleshooting.

2) Consider adding an IP address, such as in this example:

pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -s 256 --ip AAA.BBB.CCC.DDD -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.VnicTxRx.pcapng &

Above, substitute the problem IP address that being investidated, in place of AAA.BBB.CCC.DDD.
For example, enter the default gateway IP address configured for the virtual machine.

3) Consider adding a Protocol type, such as ICMP for Ping Requests and Replies, such as in this example:

pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -s 256 --ip AAA.BBB.CCC.DDD --proto 0x01 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.VnicTxRx.pcapng &

Above, we use 0x01 for the --proto option.
Although Wikipedia may not be 100% reliable, it is a useful place to start for protocol types -- see List of IP protocol numbers