How to perform rolling packet captures for vSphere, and NSX Analysis

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

• There are times when packet level inspection is necessary to identify why performance or application issues are occurring.
• These issues can have the nature of being sporadic and occurring after normal business hours making the event unpredictable.
• The challenge is to have a method of performing captures continuously without having a single massive file, and allowing it to run continuously without human intervention.
• The intent of the capture is to characterize the event base on the packets captured.
• This article will focus on the rolling packet capture method to revealing where the issue lies.  

Environment

NSX Networking
vSphere Networking

Cause

The causes are varied. 
There could be issues with the protocol in use, or environmental issue with the physical infrastructure. 

Resolution

The rolling packet capture is a combination of ptkcap-uw and tcpdump-uw.  Pktcap-uw is the packet capture tool. Its output is streamed to the tcpdump-uw command which then produces the pcap files.  Tcpdump-uw options are used to set the number of output files and limit the size or duration of each capture file.

This method is applicable to any networking analysis need for VMware products such as  vSphere, and NSX.  The object is to have a single capture running and continuously producing some number of capture files with a specific maximum file size.  Each file will reach a specified size and then roll to the next file in sequence.  When the maximum number of files is reached it will begin overwriting the files starting with the oldest file first.  This sequence will continue until the capture is terminated.

The capture files will require a directory capable of holding several gigabytes.  The capture files will consume more space than the ESXi host can provide in its /tmp directory.  A directory called "captures" will need to be created on a selected shared datastore with enough storage capacity to support the capture files.  Since this a directory on a shared datastore it will be created once.  The datastore will be shared across all of the hosts.

a) Create a Shared Directory Called "captures"

1. cd /vmfs/volumes
2. ls -l  #Select the datastore to create the captures directory
3. cd <Selected datastore>
4. mkdir captures
5. cd captures

   Placing the capture files here allows them to be downloaded using the file download option in vCenter UI.
  
b) Collecting the Switchport and Uplink Information for a VM

1. esxcli vm process list | less                     Search the list for the Virtual Machine name. Record the WID (World ID).
2. esxcli network vm port list -w <WID>              The output for the WID will have the switchport number, uplink, and VM name.
3. Record the information. 
   This data will be used in the creation of the "pktcap-uw" command.

c) Capture Commands

The command has two parts:
pktcap-uw --switchport <Switchport ID of VM> --capture VnicTx,VnicRx --ng --snaplen 150 --ip <VM IP> -o - |tcpdump-uw -W 5 -C 1000 -w <VM | Host |Component>.switchport#-X.pcap


PKTCAP-UW Command Part 1

pktcap-uw --switchport <Switchport ID of VM> --capture VnicTx,VnicRx --ng --snaplen 150 --ip <VM IP> -o - 

Explanation of Flags:
 pktcap-uw --switchport 12345678  <==Switchport ID for the VM
     --capture VnicTx,VnicRx   <==Capture point. This is the vNic of the VM
     --ng   <== capture file formating  ng  New Generation format.
     --snaplen 150   <==  Optional  The number of bytes from each packet to capture.  The 150 bytes are the  header information of the TCP packet.  No payload data.
     --ip 172.#.#.#  <== Filtering for traffic with IP as source of destination
     -o - |tcpdump-uw -W 2 -C 1 -w <.pcapng  <== pktcap-uw output stream send to tcpdump-uw command
The most commonly used capture points for rolling captures are:
 --switchport <switchport #> --capture VnicRX,VnicTx
--uplink vmnicX --capture UplinkSndKernel,UplinkRcvKernel
There are no spaces before or after the coma in "VnicRX,VnicTx"  and, "UplinkSndKernel,UplinkRcvKernel".
If there is a space an error message will be generated concerning a syntax error.

TCPDUMP-UW Command Part 2

TCPDump and TCPDump-UW are essentially the same.  The options offered by tcpdump are the same for tcpdump-uw.
The use of filters such as "host <IP Address>" as well as operators "and | or" have the same syntax.

tcpdump-uw -W 5 -C 1000 -w <VM | Host |Component>.switchport#-X.pcapng

Explanation of Flags:
 tcpdump-uw                     ⇐Second capture command for the rolling capture command line.
-W 2                                  ⇐ Number of files to create
-C 1                                  ⇐ Size of files in Megabytes
-w <File Name>.pcapng    ⇐ Output  file that is then analyzed using Wireshark

Naming Format for Capture Files

1. The file name should indicate what component was captured.
2. What the capture point was.
      - Switchport or Uplink  Example:
      - switchport-12345678
            TestVM.switchport-12345678.pcapng
      - Uplink vmnicX
            TestVM.vmnicX.pcapng

 

Additional Information

pktcap-uw --vmk vmk4 -C 1024 -G 3600 --dir 2 -o /vmfs/volumes/datastorename/Capture/Host_iSCSI_Capture-%m-%d-%H-%M-%S-%s.pcap

• Command Breakdown:
  • --vmk vmk4: Specifies vmk4 for which the packets will be captured.
  • -C 1024: This sets the packet capture buffer size to 1024 bytes per packet. Only the first 1024 bytes of each packet will be captured.
  • -G 3600: This sets the maximum capture duration to 3600 seconds (1 hour).
  • --dir 2: This specifies the direction of the capture. 2 means capturing packets that are being transmitted and received (both directions).
  • -o Specifies the output file path and name for the capture file.
  • Timestamps with the following placeholders:
    • %m: Month
    • %d: Day
    • %H: Hour
    • %M: Minute
    • %S: Second
    • %s: Microsecond timestamp for uniqueness

 

• • IP Protocol types/numbers   
  • LACP                     --ethtype 0x8809
  • ARP                       --ethtype 0x0806
  • RARP                    --ethtype 0x8035
  • AppleTalk              --ethtype 0x809b
  • Appletalk AARP    --ethtype 0x80f3
  • Novell IPX (alt)     --ethtype 0x8137
  • Novell                   --ethtype 0x8138
  • IPv6                      --ethtype 0x86DD
  • SNMP                   --ethtype 0x814C
  • Point-to-Point        --ethtype 0x880B
  • MPLS (unicast).    --ethtype 0x8847    
  • MPLS (multicast)  --ethtype 0x8848    
  • PPPoE(Discovery) --ethtype 0x8863    
  • PPoE (Session)  --ethtype 0x8864   
  • NLB                       --ethtype 0x886F    
  • LLDP                     --ethtype 0x88CC    

Example:  Packet Capture on Vsan VMK, ICMP traffic from source IP, captured to file.  Command: # pktcap-uw --vmk vmk2 --proto 0x01 --ip 192.#.#.# -o /vmfs/volumes/vmk.pcap