Packet Capturing for ESXi and NSX Testing and Troubleshooting

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

There are times when capturing packets is a necessary and logical next step.  Examples are inconsistent network connectivity between a physical device and a virtual machine. A security camera sends video to a recorder or session manager for storage or display clients.  The image as seen by the user is less than expected and experiencing video artifacts.  Another example may be that network performance may have recently become slower.  Analyzing the packets at strategic points in the data path will often reveal the issue.

Environment

NSX 3.x  NSX 4.x
ESXi  6.7U2  ESXi 7    ESXi 8

Cause

Latency has two possible locations where it can be rooted.  It can be application latency or network latency.  To define which it is, simultaneous capture must be done at the source and destination.  Then the packet's time stamps can be compared.  If a packet leaves the source then within just a few milliseconds arrives at the destination the network data path is cleared.  The destination then generates a response three minutes later and sends it to the source.  The timing from destination to source (reverse direction) has the same latency of just  a few milliseconds

The packet analysis will expose this issue if and only if the two captures are done simultaneously.  The strategy is to compare two points against each other for the same network flow.

Similarly the two point capture can discern that the issue is the data path and not the Application.  The latency could be unidirectional.  This would then be network latency.  Packet captures will expose which it is by careful comparison and analysis.   

Resolution

First determine which source and destination hosts are to be used for the analysis.
These nodes are typically virtual machines or, physical hosts, or the ESXi host.

The command pktcap-uw is utilized for the vast majority of testing. This command is native to the ESXi host.  It has several options for capture points. 

The command is most often used with the following options.

pktcap-uw -- switchport 12345678 --capture VnicTx,VnicRx --ng --snapLen 150 -o <filename.pcapng>
This will capture packets at the closest point to the virtual machine.  This is the expected method of virtual machine packet capture file creation.  The --snaplen 150 gathers the first 150 bytes of the packet.  The TCP headers are all in the first 150 bytes of the packet.  The data payload of the packet is not needed for support to analyze.  Not capturing the payload data allows the capture to consume space with only the data needed for analysis.

The --ng option is the format of the capture.  This is the newest version of the file format which gathers considerably more information than the standard pcap format.  This format will be required by support.  The extension of the file will be .pcapng.  The naming convention will contain information of the capture point such that it is easily discernible where the location is.

Switchport Naming:
VM_Name.swprt#-X.pcapng
The VM name may be abbreviated if needed.  This abbreviation must be noted where support can match the abbreviation to the true virtual machine name. The -X starts at 1 and increments for each new iteration if a series of captures is to be done.

pktcap-uw -- uplink vmnicX --capture UplinkSndKernel,UplinkRcvKernel --ng --snapLen 150 -o <filename.pcapng>
This command captures packets at the last point before the ESXi host places the packet on the physical wire.  This is how local VM to remote destination, or remote destination to local VM traffic is proven to exist.  

Uplink filename format:
vmnicX.host.-X.pcapng

vmnicX is the name of the uplink.  Example vmnic1 or eth0
Host is the name of the ESX host or Edge node. The -X starts at 1 and increments for each new iteration if a series of captures is to be done.

 

The remote destination is any physical or virtual host that must traverse the edge device to enter the virtual environment.  

Careful consideration should be given to the location of where the capture files will be placed.  For the ESX host creating a capture file on a shared datastore should be used.  The datastore will be located in the /vmfs/volumes directory.  Change directories to this location and then decide which datastore to create a directory called "captures."
cd /vmfs/volumes
cd <datastore>
mkdir captures
From this location the capture commands will be executed.  The files will be created here.

Network Latency
At this point the capture commands should be ready to execute on both the local and remote nodes(Hosts).
Start the Captures
Perform the task that will generate the traffic to be analyzed. 

The traffic is generated by performing the relevant task that is problematic such as Logins, Sql queries, …etc.  

Download the capture file from both source and destination.  Open them up in Wireshark to compare the times that the packet left the source and arrived at the destination to when the packet arrived at the destination.  Then compare the elapsed time to the elapsed time that the response from the destination is sent back to the source.

No network latency
Source ⇒Destination   3mSec    13:00:01.12134
Destination ⇒ Source  4mSec    13:05:4.123
The time on the wire was similar and measured in mSec for both directions.  This would not be network latency.
The time the response was sent was 4 minutes.  This would be the definition of application latency.  The packet for the response was not generated quickly, but once it was generated the packet time on the wire was fast. 

Network latency
Source ⇒Destination   3mSec    13:00:01.12134
Destination ⇒ Source  50mSec  13:00:03.123
The source made its request and the destination responded in 2 mSec.  However, it took the response 50 mSec to be received at the source.  In one direction the travel time was shorter than the return. The generation of the response packet was 2 mSec.  This is the definition of network latency.  In this example the latency was seen in one direction.  Network latency can manifest in either direction or both directions at the same time.

Connectivity Testing
The same setup done for latency testing is also used for connectivity verification.  In this case the goal is to determine if a packet sent arrives at the destination, and then verify that the response packet arrives at the source.  The timing of the packets is not the focus. The focus of this task is to prove that packet 0x23a1  arrived at the destination. The destination sends a response packet 0xaa35 and does it arrive at the source.

 

Additional Information

To display a pcap for quick verification of what was capture the following command can be appended to the capture command.
-o - |tcpdump-uw –enr - -n   <CapterFile.pcapng>|less

Example
pktcap-uw -- switchport 12345678 --capture VnicRx,VnicRx --ng --snapLen 150 -o - |tcpdump-uw –enr - -n   <CapterFile.pcapng>|less

The capture command for the switchport is used for any virtual machine production or management (Edge Device).  There are very few times where a capture from within the Edge device or VM is truly needed. If a capture from within the Edge is needed then Broadcom support will be needed. Internal captures do have a performance impact on the edge.