Using the pktcap-uw tool in ESXi 5.5 and later
search cancel

Using the pktcap-uw tool in ESXi 5.5 and later

book

Article ID: 341568

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

The pktcap-uw tool is an enhanced packet capture and analysis tool that can be used in place of the legacy tcpdump-uw tool. The pktcap-uw tool is included by default in ESXi 5.5 and later versions. This article provides information on using the pktcap-uw tool for packet capture and analysis.

Note: The tcpdump-uw tool can only capture packets/frames at the vmkernel interface level and cannot capture frames at the uplinks, or vSwitch, or virtual port levels. The new pktcap-uw tool allows traffic to be captured at all points within the hypervisor for greater flexibility and improved troubleshooting.


Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Resolution

 Using the pktcap-uw tool

Note:
  1. pktcap defaults to inbound traffic only.
  2. In vSphere 6.5 and earlier, specify the direction of traffic using --dir 0 for inbound and --dir 1 for outbound. You can’t specify traffic going both ways at the same time. However, in vSphere 6.7 and later, you can specify the direction of traffic using --dir 0 for inbound, --dir 1 for outbound, or --dir 2 for both.
  3. Two (or more) separate traces can be run in parallel but need to be merged later in Wireshark.
 
  • To obtain basic help and syntax information, use the -h option:

    # pktcap-uw -h |more
     
  • To view a live capture of a vmkernel ports traffic:

    # pktcap-uw --vmk vmkX

    For example, to capture frames/packets on vmk0:

    # pktcap-uw --vmk vmk0
     
  • To view a live capture of a specific physical network card on the host vmnic:

    # pktcap-uw --uplink vmnicX

    For example, to capture frames/packets on vmnic7:

    # pktcap-uw --uplink vmnic7
     
  • To view a live capture of a particular vSwitch port for a virtual machine, use the --switchport option:

    # pktcap-uw --switchport switchportnumber

    For example, to capture frames or packets to and from a virtual machine connected to dvSwitchport 8:

    # pktcap-uw --switchport 8
     
  • To capture the output to a file, use -o option:

    # pktcap-uw --vmk vmk# -o file.pcap

    For example, to capture the packets from vmk0 and save to test.pcap file under /tmp directory:

    # pktcap-uw --vmk vmk0 -o /tmp/test.pcap

    Notes:
    • To display the output of the pktcap-uw live in the terminal in a more readable format append this to the command:  -o - |tcpdump-uw -ne -c 10
      Commnad Example:

      pktcap-uw --uplink vmnic0 --capture UplinkSndKernel,UplinkRcvKernel --ng --snap 150 -o - |tcpdump-uw -ne -c 10
      tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
      listening on vmk0, link-type EN10MB (Ethernet), capture size 262144 bytes
      20:19:01.448472 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 262: x.x.x.32.22 > x.x.x.10.61723: Flags [P.], seq 3923134684:3923134892, ack 3124475961, win 128, length 208
      20:19:01.448895 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 326: x.x.x.32.22 > x.x.x.10.61723: Flags [P.], seq 208:480, ack 1, win 128, length 272
      20:19:01.449443 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 294: x.x.x.32.22 > x.x.x.10.61723: Flags [P.], seq 480:720, ack 1, win 128, length 240
      20:19:01.449621 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 285: x.x.x.32.50260 > x.x.x.200.514: SYSLOG local6.info, length: 243
      20:19:01.449696 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 294: x.x.x.32.22 > x.x.x.10.61723: Flags [P.], seq 720:960, ack 1, win 128, length 240
      20:19:01.450799 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 278: x.x.x.32.22 > x.x.x.10.61723: Flags [P.], seq 960:1184, ack 1, win 128, length 224
      20:19:01.450827 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 220: x.x.x.32.50260 > x.x.x.200.514: SYSLOG local6.info, length: 178
      20:19:01.450841 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 294: x.x.x.32.22 > x.x.x.10.61723: Flags [P.], seq 1184:1424, ack 1, win 128, length 240
      20:19:01.450935 12:hh:gg:ff:dd:ss > aa:bb:dd:ff:gg:hh:12, ethertype IPv4 (0x0800), length 60: x.x.x.10.61723 > x.x.x.32.22: Flags [.], ack 208, win 4106, length 0
      20:19:01.451469 aa:bb:dd:ff:gg:hh:12 > 12:hh:gg:ff:dd:ss, ethertype IPv4 (0x0800), length 630: x.x.x.32.22 > x.x.x.10.61723: Flags [P.], seq 1424:2000, ack 1, win 128, length 576
      10 packets captured
      10 packets received by filter
      0 packets dropped by kernel
      The name of the uplink is vmnic0.
      The session capture point is UplinkSndKernel,UplinkRcvKernel.
      The output file format is pcapng.
      The snap len is 150.
      pktcap: The output file is -.
      pktcap: No server port specifed, select 6238 as the port.
      pktcap: Local CID 2.
      pktcap: Listen on port 6238.
      pktcap: Main thread: 502433561472.
      pktcap: Dump Thread: 502434096896.
      pktcap: Join with dump thread failed.
      pktcap: Destroying session 18.
      pktcap:
      pktcap: Dumped 0 packet to file -, dropped 0 packets.
      pktcap: Done.
      This command caught 10 frames and the first 150 bytes of each packet. 
      -c  Number of packets to capture  (tcpdump-uw)
      --snap  length of the packet to capture  (pktcap-uw)
      -n  do not resolve IP address  (tcpdump-uw)
      -e print MAC layer header (MAC addresses)
    • When you are using the -o option, output is in pcap format regardless of other settings.
    • To specify how many seconds pktcap-uw will run, use -G <seconds>.
    • To specify pktcap-uw to run only until the output is a certain file size (in megabytes), use -C <file_size>.
    • To end the capture, ensure to use Ctrl-C multiple times instead of Ctrl-Z because Ctrl-Z may leave background processes running that may prevent subsequent pktcap-uw commands from running and report the error:

      error: Can't create the session, Exiting


    • When performing captures of the vmnic or vnic the preferred capture points are:
      --capture VnicRx,VnicTx
      --capture UplinkRcvKernel,UplinkSndKernel
      Command example:
      pktcap-uw --uplink vmnic1 --capture UplinkRcvKernel,UplinkSndKernel --ng
      pktcap-uw --switchport 12345687 --capture VnicRX,VnicTx --ng

Capture points

There are 2 points at which a traffic shall be captured in vSwitch i.e. switchport and uplink. Switchport connects to all the vNIC and vmk adapters and uplink is the actual physical NIC (vmnic). We will first identify the capture point based on the issue and apply the command syntax accordingly. In the below diagram you shall see the architecture how VM’s, kernel and physical adapters are connected. 

Identifying the active uplink carrying a specific vm/vmk traffic.

  • Run the command "net-stats -l" and this will list the switchports that are mapped to vm/vmk.

  • Option 1 to identify the uplink: Run the command "esxtop" and press "n". This will show the mapping between switchport and uplink.

  • Option 2 to identify the uplink: Run the command "vsish" and press enter. "cat/portsets/vSwitch_name/ports/switchport_number/teamuplink"

Note: For vSwitch name and switchport number refer to "net-stats -l" output.

  • Option 3 to identify the uplink: Run the command "netdbg vswitch instance list" and press enter.

Advanced Usage: trace multiple ports at the same time

As an example, trace a particular vSwitch port and its associated uplink at the same time:

  1. To get the vSwitch port number, run this command:

    net-stats -l

  2. Identify and make a note these parameters:

    Port ID returned by the esxtop command — --switchport 12345678

    vmnic2 physical port that you want to trace — --uplink vmnic2 -

    location of the output pcap file — /tmp/vmnic2.pcap

  3. Run the pktcap-uw command to capture packets at both points simultaneously:

    pktcap-uw --switchport 12345678 -o /tmp/12345678.pcap & pktcap-uw --uplink vmnic2 -o /tmp/vmnic2.pcap &

    Note: The command shell we assume here is the Bash shell. Therefore, if you have more than one uplink on the vswitch, you just add the command after the ending & and add a & at the end.
     
  4. You can stop pktcap-uw tracing with the kill command:

    kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)
     
  5. Run this command to check that all pktcap-uw traces are stopped:

    lsof |grep pktcap-uw |awk '{print $1}'| sort -u



Additional Information

Packet capture analysis is typically conducted in a live environment when issues persist. Please note that packet capture data is not stored in ESXi logs. If you require additional assistance with troubleshooting, we recommend reaching out to Broadcom support by creating a support case. Please refer the KB - https://knowledge.broadcom.com/external/article/324495
Powered by Wolken Software