Packet capture on ESXi using the pktcap-uw tool

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on using the pktcap-uw tool for packet capture and analysis.

 

The pktcap-uw tool is an enhanced packet capture and analysis tool that can be used in place of the legacy tcpdump-uw tool and is included by default in ESXi 5.5 and later versions. This tool allows traffic to be captured at all points within the hypervisor for greater flexibility and improved troubleshooting.

Environment

VMware vSphere ESXi

Resolution

As a starting point, refer to ESXi Network Troubleshooting Tools, specifically to the section titled "Pktcap-uw".

Under that title, look for the following graphic:

 

Packet captures are done by entering commands at the command line of a session via SSH (or Server Console access via KVM -- “Keyboard, Video (monitor), Mouse.”) with root access privileges. 

Capture Points

There are a variety of points at which a traffic can be captured, depending on the type of switch in use. For example, standard vSwitch or a Distributed Virtual Switch (DVS).

Capture output can be directed to the user's screen and displayed (by parsing the output with the tcpdump-uw tool), or it can be directed to a datastore which can then be downloaded and analyzed with a tool such as Wireshark.  

Plan on directing the output to a datastore, NEVER use the /tmp directory as a place to put the data.

• Instead, first determine what datastores are available to the ESXi host using the command:
  
  df -h
  
  
• Then, change to the datastore:
  
cd /vmfs/volumes/FULL_PATH_TO_DATASTORE
  
  Note: When sending the data to a VSAN datastore (though it is recommended to use a non vSAN datastore), enclose the command in quotes, such as:
  
  cd "/vmfs/volumes/FULL_PATH_TO_DATASTORE"
  
  
• As a general rule, when doing captures from multiple ESXi hosts, use a datastore that is accessible from all of those hosts.
• It is also a good idea to create a sub-folder or sub-directory at that datastore. For example, "CASE_12345678" where instead of "12345678", use the Broadcom Support Case Number.  
  
  Example:
  
  cd "/vmfs/volumes/FULL_PATH_TO_DATASTORE"
mkdir Case_12345678

Standard vSwitch Capture Points

In vSwitch, i.e. switchport and uplink.

Switchports connect to all the vNIC, vmk adapters and uplinks (which are the actual physical NIC (vmnic)).

First identify the capture point based on the issue and apply the command syntax accordingly.

In the below diagram note the architecture and how VMs, kernel and physical adapters are connected. 

Identifying the active uplink carrying a specific vm/vmk traffic.

• Run the command "net-stats -l" and this will list the switchports (listed under the PortNum column) that are mapped to vm/vmk.

• Option 1 to identify the uplink: run the command "esxtop" and press "n". This will show the mapping between switchport (listed under the PORT-ID column) and uplink (under the column TEAM-PNIC).

• Option 2 to identify only the uplink: run the command "vsish" and press enter. "cat /net/portsets/vSwitch_name/ports/switchport_number/teamUplink". 

Note: For vSwitch name and switchport number, refer to "net-stats -l" output above.

• Option 3 to identify the uplink and switchport number: run the command "netdbg vswitch instance list" and press enter.

Advanced Usage: trace multiple ports at the same time

As an example, trace a particular vSwitch port and its associated uplink at the same time:

1. To get the vSwitch port number:
   
   net-stats -l

2. Identify and make a note of these parameters:

   Port ID returned by the esxtop command — --switchport <switchport-id>

   vmnic# physical port to trace — --uplink vmnic2 -

   Location of the output pcap file — /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/

3. Run the pktcap-uw command to capture packets at both points simultaneously, ensure to replace the ####### with the specific switchport that the capture is going to be taken on.:
   
   pktcap-uw --switchport ######## -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_#/esxi01.switchport.########.pcapng & pktcap-uw --uplink vmnic# -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi.uplink.vmnic#.pcapng &

Note: The command shell we assume here is the Bash shell. Therefore, if there is more than one uplink on the vSwitch, just add the command after the ending & and add a & at the end.
   -When doing only a single capture, omit the "&"
   -To stop a single capture, press and hold the "CTRL" key and touch "C".
   -Expect to see a number of messages on the screen as the packets are being captured.
   -For better viewing while this is happening, start a duplicate session using SSH. 
4. To stop pktcap-uw tracing with the kill command:
   
   kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)
    
5. Run this command to check that all pktcap-uw traces are stopped:
   
   lsof |grep pktcap-uw |awk '{print $1}'| sort -u

Distributed Virtual Switch (DVS) Capture Points

Per the diagram, there are different capture points.

1. The closest capture point to the physical infrastructure (i.e. the entry / exit point between the ESXi hypervisor and the physical infrastructure) is called the --uplink capture point.
   
   Example for directing the output to a file is shown below, where the traffic gathered is both the sent (UplinkSndKernel) and received (UplinkRcvKernel) traffic:
   
   pktcap-uw --uplink vmnic# --capture UplinkSndKernel,UplinkRcvKernel -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_#/esxi.uplink.vmnic#.UplinkSndRcvKernel.pcapng &

Example for directing the output to the screen, traffic gathered with this command is also capturing both sent and received traffic:
   
   pktcap-uw --uplink vmnic# --capture UplinkSndKernel,UplinkRcvKernel -o - | tcpdump-uw -r - -enn
   
   Note:
   
   To terminate the capture when directing the output to a file:
   
   kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)
   
   To terminate the capture when directing the output to the screen, press and hold the CTRL key and touch "C".
   
   
2. The capture point closest to the actual virtual machine's virtual NIC is called the --switchport capture point.
   • Example for directing the output to a file:
     
     pktcap-uw --switchport ######## --capture VnicTx,VnicRx -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi.switchport.########.VnicTxRx.pcapng &

   • Example for directing the output to the screen:
     
     pktcap-uw --switchport ######## --capture VnicTx,VnicRx -o - | tcpdump-uw -r - -enn
     
     
3. The other two capture points are shown in the OVERVIEW diagram at the top of this RESOLUTION section, and generally are only used when working with a Broadcom Technical Support Engineer (TSE) during an investigation via a Broadcom Support Case. 

Additional Information

Please note the packet capture data is not stored in ESXi logs and that it is not recommended to run multiple instances of packet captures for long periods of time unless directed to do so by VMware by Broadcom support. 

If additional assistance is needed with troubleshooting, we recommend reaching out to Broadcom support by creating a support case using the instructions at Creating and managing Broadcom support cases.

The way to capture data in .pcapng files, which can be then loaded into Wireshark for detailed analysis, would be with the -o parameter of the pktcap-uw command.

 

NOTES REGARDING FILES WHERE DATA IS CAPTURED:

1. NEVER use the /tmp directory to store .pcapng data using the -o parameter of the pktcap-uw command.
2. Instead, ideally identify a VMFS datastore. If possible, avoid the use of a VSAN datastore. A fibre channel, iSCSI or NFS datastore, or even a local disk datastore are better choices.
3. Within the datastore, it is recommented to create a self-descriptive folder name such as "Packet_Captures". If needed and are working with Broadcom Support on an investigation, a good choice would be "Case_####" where "##### is the Support Case number. 
4. Within that folder, use self-descriptive filenames, such as "HOSTNAME.VMVnicName.CapturePointNames.pcapng". 
   • An example might be "HOSTNAME.VMNAME.eth0.VnicTxRx.pcapng" (assuming in this case that were doing a --switchport capture using VnicTx,VnicRx as the --capture option.

TIPS TO HELP KEEP PACKET CAPTURE FILES SMALLER - TO MAKE ANALYSIS AND COLLABORATION EASIER AND FASTER:

1. If no payload is required to be captured, add the following. For example, a packet size limiting switch, as per this example:
   
   pktcap-uw --switchport ######## --capture VnicTx,VnicRx -s 256 -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi.switchport.########.VnicTxRx.pcapng &
   
   In the above example, the addition of the -s 256 switch will limit the size of each packet captured to the first 256 bytes, which contains the header, which is usually the most important info when troubleshooting.
   
   
2. Consider adding an IP address, such as in this example:
   
   pktcap-uw --switchport ######## --capture VnicTx,VnicRx -s 256 --ip ###.###.###.### -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi.switchport.########.VnicTxRx.pcapng &

In the above example, substitute the problem IP address that being investigated, in place of ###.###.###.###. For example, enter the default gateway IP address configured for the virtual machine. 
   
   
3. NOTE:  In the above example, if the problem IP address that being investigated belongs to a virtual machine where the traffic is being carried by NSX, then the command would be modified like the following:

pktcap-uw --switchport ######## --capture VnicTx,VnicRx -s 256 --rcf "geneve and host ###.###.###.###" -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi.switchport.########.VnicTxRx.pcapng &

WHERE:  substitute the problem IP address that being investigated, in place of ###.###.###.###.  

Additional filters can be used, such as ICMP for ping requests and replies, however it is only recommended to use filters in specific instances to avoid over filtering. Please note the --proto 0x01 is only used as an example in the command below:

 pktcap-uw --switchport ######## --capture VnicTx,VnicRx -s 256 --ip ###.###.###.### --proto 0x01 -o  /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/esxi.switchport.########.VnicTxRx.pcapng &

RELATED LINKS :

To provide advice and best practices when using the pktcap-uw tool when the ESXi hosts are heavily loaded, see Considerations to run pktcap-uw under heavy network load.