Packet capture on ESXi using the pktcap-uw tool

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on using the pktcap-uw tool for packet capture and analysis.

 

The pktcap-uw tool is an enhanced packet capture and analysis tool that can be used in place of the legacy tcpdump-uw tool and is included by default in ESXi 5.5 and later versions. 

Note: The new pktcap-uw tool allows traffic to be captured at all points within the hypervisor for greater flexibility and improved troubleshooting.

Resolution

As a starting point, refer to ESXi Network Troubleshooting Tools, specifically to the section titled "Pktcap-uw".

Under that title, look for the following graphic:

 

Packet captures are done by entering commands at the command line of a session via SSH (or Server Console access via KVM -- “Keyboard, Video (monitor), Mouse.”) with root access privileges. 

Capture Points

There are a variety of points at which a traffic can be captured, depending on the type of switch in use. For example, standard vSwitch or a Distributed Virtual Switch (DVS).

Capture output can be directed to the user's screen and displayed (by parsing the output with the tcpdump-uw tool), or it can be directed to a disk file which can then be downloaded and analyzed with a tool such as Wireshark.  

Plan on directing the output to a disk file, NEVER use the /tmp directory as a place to put the data.

• Instead, first determine what datastores are available to the ESXi host using the command:
  
  df -h
  
  
• Then, change to the datastore:
  
cd /vmfs/volumes/FULL_PATH_TO_DATASTORE
  
  Note: When sending the data to a VSAN datastore, enclose the command in quotes, such as:
  
  cd "/vmfs/volumes/FULL_PATH_TO_DATASTORE"
  
  
• As a general rule, when doing captures from multiple ESXi hosts, use a datastore that is accessible from all of those hosts.
• It is also a good idea to create a sub-folder or sub-directory at that datastore. For example, "CASE_12345678" where instead of "12345678", use the Broadcom Support Case Number.  
  
  Example:
  
  cd "/vmfs/volumes/FULL_PATH_TO_DATASTORE"
mkdir Case_12345678

Standard vSwitch Capture Points

In vSwitch, i.e. switchport and uplink.

Switchport connects to all the vNIC and vmk adapters and uplink is the actual physical NIC (vmnic).

We will first identify the capture point based on the issue and apply the command syntax accordingly.

In the below diagram note the architecture and how VMs, kernel and physical adapters are connected. 

Identifying the active uplink carrying a specific vm/vmk traffic.

• Run the command "net-stats -l" and this will list the switchports that are mapped to vm/vmk.

• Option 1 to identify the uplink: run the command "esxtop" and press "n". This will show the mapping between switchport and uplink.

• Option 2 to identify the uplink: run the command "vsish" and press enter. "cat /net/portsets/vSwitch_name/ports/switchport_number/teamUplink"

Note: For vSwitch name and switchport number, refer to "net-stats -l" output.

• Option 3 to identify the uplink: run the command "netdbg vswitch instance list" and press enter.

Advanced Usage: trace multiple ports at the same time

As an example, trace a particular vSwitch port and its associated uplink at the same time:

1. To get the vSwitch port number:
   
   net-stats -l

2. Identify and make a note of these parameters:

   Port ID returned by the esxtop command — --switchport 12345678

   vmnic2 physical port to trace — --uplink vmnic2 -

   Location of the output pcap file — /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/

3. Run the pktcap-uw command to capture packets at both points simultaneously:
   
   pktcap-uw --switchport 67108879 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.pcapng & pktcap-uw --uplink vmnic2 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.uplink.vmnic2.pcapng &

Note: The command shell we assume here is the Bash shell. Therefore, if there is more than one uplink on the vSwitch, just add the command after the ending & and add a & at the end.
   
   
4. To stop pktcap-uw tracing with the kill command:
   
   kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)
    
5. Run this command to check that all pktcap-uw traces are stopped:
   
   lsof |grep pktcap-uw |awk '{print $1}'| sort -u

Notes:

• When doing only a single capture, omit the "&"
• To stop a single capture, press and hold the "CTRL" key and touch "C".
• Expect to see a number of messages on the screen as the packets are being captured.
• For better viewing while this is happening, start a duplicate session using SSH. 

Distributed Virtual Switch (DVS) Capture Points

Per the diagram, there are different capture points.

1. The closest capture point to the physical infrastructure (i.e. the entry / exit point between the ESXi hypervisor and the physical infrastructure is called the --uplink capture point.
   
   Example for directing the output to a file:
   
   pktcap-uw --uplink vmnic2 --capture UplinkSndKernel,UplinkRcvKernel -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.uplink.vmnic2.UplinkSndRcvKernel.pcapng &

Example for directing the output to the screen:
   
   pktcap-uw --uplink vmnic2 --capture UplinkSndKernel,UplinkRcvKernel -o - | tcpdump-uw -r - -enn
   
   Note:
   
   To terminate the capture when directing the output to a file:
   
   kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)
   
   To terminate the capture when directing the output to the screen, press and hold the CTRL key and touch "C".
   
   
2. The capture point closest to the actual virtual machine's virtual NIC is called the --switchport capture point.
   • Example for directing the output to a file:
     
     pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.VnicTxRx.pcapng &

   • Example for directing the output to the screen:
     
     pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -o - | tcpdump-uw -r - -enn
     
     
3. The other two capture points are shown in the OVERVIEW diagram at the top of this RESOLUTION section, and generally are only used when working with a Broadcom Technical Support Engineer (TSE) during an investigation via a Broadcom Support Case. 

Additional Information

Please note the packet capture data is not stored in ESXi logs.

 

The way to capture data in .pcapng files, which can be then loaded into Wireshark for detailed analysis, would be with the -o parameter of the pktcap-uw command.

 

NOTES REGARDING FILES WHERE DATA IS CAPTURED:

1. NEVER use the /tmp directory to store .pcapng data using the -o parameter of the pktcap-uw command.
2. Instead, ideally identify a VMFS datastore. If possible, avoid the use of a VSAN datastore. A fibre channel, iSCSI or NFS datastore, or even a local disk datastore are better choices.
3. Within the datastore, it is recommented to create a self-descriptive folder name such as "Packet_Captures". If needed and are working with Broadcom Support on an investigation, a good choice would be "Case_####" where "##### is the Support Case number. 
4. Within that folder, use self-descriptive filenames, such as "HOSTNAME.VMVnicName.CapturePointNames.pcapng". 
   • An example might be "HOSTNAME.VMNAME.eth0.VnicTxRx.pcapng" (assuming in this case that were doing a --switchport capture using VnicTx,VnicRx as the --capture option.

If additional assistance is needed with troubleshooting, we recommend reaching out to Broadcom support by creating a support case using the instructions at Creating and managing Broadcom support cases.

 

TIPS TO HELP KEEP PACKET CAPTURE FILES SMALLER - TO MAKE ANALYSIS AND COLLABORATION EASIER AND FASTER:

1. If no payload is required to be captured, add the following. For example, a packet size limiting switch, as per this example:
   
   pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -s 256 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.VnicTxRx.pcapng &A
   
   In the above example, the addition of the -s 256 switch will limit the size of each packet captured to the first 256 bytes, which contains the header, which is usually the most important info when troubleshooting.
   
   
2. Consider adding an IP address, such as in this example:
   
   pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -s 256 --ip ###.###.###.### -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.VnicTxRx.pcapng &

In the above example, substitute the problem IP address that being investigated, in place of ###.###.###.###. For example, enter the default gateway IP address configured for the virtual machine. 
   
   
3. Consider adding a protocol type, such as ICMP for Ping Requests and Replies. For example:
   
   pktcap-uw --switchport 67108879 ---capture VnicTx,VnicRx -s 256 --ip ###.###.###.### --proto 0x01 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi01.switchport.67108879.VnicTxRx.pcapng &

In the above example, we use 0x01 for the --proto option.

Although Wikipedia may not be 100% reliable, it is a useful place to start for protocol types -- see List of IP protocol numbers.

RELATED LINKS :

To provide advice and best practices when using the pktcap-uw tool when the ESXi hosts are heavily loaded, see Considerations to run pktcap-uw under heavy network load.