Performing Rolling Packet Captures on ESXi for High-Volume Traffic

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on using the pktcap-uw tool in conjunction with tcpdump-uw to perform rolling packet captures. This method is specifically designed for scenarios where the volume of traffic is large and cannot be effectively filtered, or when an intermittent issue requires the capture to run for an extended period without generating a massive file.

Large single packet capture files present significant challenges for administrators. Single files exceeding several gigabytes frequently cause analysis tools like Wireshark to crash or become unresponsive due to memory exhaustion. Furthermore, these large traces are difficult to manage as they are slow to upload to Support and cumbersome to download for local investigation. 

Environment

VMware vSphere ESXi

Resolution

Ensure root access to the ESXi shell via SSH or console is present.

• Identify the capture points based on the issue. On standard vSwitch, the capture points are switchport and uplink. Switchport information can be extracted using net-stats -l command. This command will list the switchports (listed under the PortNum column) that are mapped to vm/vmk.
  
  
  
• Uplink information can be extracted by running the command "esxtop" and pressing "n". This will show the mapping between switchport (listed under the PORT-ID column) and uplink (under the column TEAM-PNIC).
  
  
  
• As the goal is to save the packet capture to a file. Plan on directing the output to a datastore, NEVER use the /tmp directory as a place to put the data. Use df -h to determine what datastores are available and how much space is available on them.
  
  
• Once the information has been collected, use the following commands to run the bi-directional rolling capture:
  
  pktcap-uw --switchport <switchport-id> --capture VnicRx,VnicTx --ng -o - | tcpdump-uw -en -W <number of files> -C <Size of file (1000=1GB)> -w host-switchport-rolling.pcapng

pktcap-uw --uplink vmnicX --capture UplinkSndKernel,UplinkRcvKernel --ng -o - | tcpdump-uw -en -W <number of files> -C <Size of file (1000=1GB)> -w host-uplink-rolling.pcapng

• To stop a single capture, press and hold the "CTRL" key and touch "C". To stop pktcap-uw tracing with the kill command:
  
  kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)

• The files will be saved as host-uplink-rolling.pcapng0, host-uplink-rolling.pcapng1 and so on. They can be combined later or analyzed as a single file as well.

Additional Information

When using the rolling capture method, you can calculate the total disk space footprint by multiplying the number of files by the size of each file. In the example provided, 10 files at 500 MB each ensures the capture will never exceed 5 GB of disk space. If host CPU utilization becomes a concern during high-volume sessions, you may consider limiting the snapshot length by adding -s 256 to the pktcap-uw portion of the command, which captures only the first 256 bytes of each packet and discards the heavier payload data.

Packet capture on ESXi using the pktcap-uw tool