ESXi and NSX: Packet Capture for Network Latency and connectivity Troubleshooting

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

There are times when capturing packets is a necessary and logical next step. Examples are inconsistent network connectivity between a physical device and a virtual machine. A security camera sends video to a recorder or session manager for storage or display clients. The image as seen by the user is less than expected and experiencing video artifacts. Another example may be that network performance may have recently become slower. Analyzing the packets at strategic points in the data path will often reveal the issue.

Symptoms that may be observed

Retransmissions on overlay networks may be seen
Intermittent latency of 100-500ms between Edge and Host Components
Latency isolated to physical network hops (e.g., Switch issues)

Environment

VMware NSX
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Latency has two possible locations where it can be rooted. It can be application latency or network latency. To define which it is, simultaneous capture must be done at the source and destination. Then the packet's time stamps can be compared. If a packet leaves the source then within just a few milliseconds arrives at the destination the network data path is cleared. The destination then generates a response three minutes later and sends it to the source. The timing from destination to source (reverse direction) has the same latency of just a few milliseconds

The packet analysis will expose this issue if and only if the two captures are done simultaneously. The strategy is to compare two points against each other for the same network flow.

Similarly the two point capture can discern that the issue is the data path and not the Application. The latency could be unidirectional. This would then be network latency. Packet captures will expose which it is by careful comparison and analysis.

Resolution

First determine which source and destination hosts are to be used for the analysis.
Perform the following steps in both the Source and destination of the VMware ESXi host.
1. 1. On an ESXi host NOT prepared for NSX, to see both the switchport (<switchport-id>) and uplink( <vmnic_number>), run following command on ESXCLI: netdbg vswitch instance list
  2. On an ESXi host prepared for NSX, to see both the switchport and uplink, run following command on ESXCLI: nsxdp-cli vswitch instance list

The command is most often used with the following options, which captures at Vnic switchport of the VM.

pktcap-uw --switchport <switchport_id>--capture VnicTx,VnicRx --ng --snapLen 150 -o <filename.pcapng>

This will capture packets at the closest point to the virtual machine. This is the expected method of virtual machine packet capture file creation. The --snaplen 150 gathers the first 150 bytes of the packet. The TCP headers are all in the first 150 bytes of the packet. The data payload of the packet is not needed for support to analyze. Not capturing the payload data allows the capture to consume space with only the data needed for analysis.
The --ng option is the format of the capture. This is the newest version of the file format which gathers considerably more information than the standard pcap format. This format will be required by support. The extension of the file will be .pcapng. The naming convention will contain information of the capture point such that it is easily discernible where the location is.
Switchport Naming: VM_Name.swprt#-X.pcapng, The VM name may be abbreviated if needed. This abbreviation must be noted where support can match the abbreviation to the true virtual machine name. The -X starts at 1 and increments for each new iteration if a series of captures is to be done.

pktcap-uw --uplink vmnicX --capture UplinkSndKernel,UplinkRcvKernel --ng --snapLen 150 -o <filename.pcapng>

This command captures packets at the last point before the ESXi host places the packet on the physical wire. This is how local VM to remote destination, or remote destination to local VM traffic is proven to exist.
Uplink filename format: vmnicX.host.-X.pcapng, where vmnicX is the name of the uplink. Example vmnic1. Host is the name of the ESX host or Edge node. The -X starts at 1 and increments for each new iteration if a series of captures is to be done.
The remote destination is any physical or virtual host that must traverse the edge device to enter the virtual environment.

Careful consideration should be given to the location of where the capture files will be placed. For the ESX host creating a capture file on a shared datastore should be used. The datastore will be located in the /vmfs/volumes directory. Change directories to this location and then decide which datastore to create a directory called "captures". From this location the capture commands will be executed. The files will be created here as below.
cd /vmfs/volumes
cd <datastore>
mkdir captures
Network Latency
At this point the capture commands should be ready to execute on both the local and remote nodes(Hosts).
Start the Captures and Perform the task that will generate the traffic to be analyzed.
Download the capture file from both source and destination. Open them up in Wireshark to compare the times that the packet left the source and arrived at the destination to when the packet arrived at the destination. Then compare the elapsed time to the elapsed time that the response from the destination is sent back to the source.

No network latency
```
Source ⇒Destination   3mSec    13:00:01.12134
Destination ⇒ Source  4mSec    13:05:4.123
```
The time on the wire was similar and measured in mSec for both directions. This would not be network latency.
The time the response was sent was 4 minutes. This would be the definition of application latency. The packet for the response was not generated quickly, but once it was generated the packet time on the wire was fast.
Network latency
```
Source ⇒Destination   3mSec    13:00:01.12134
Destination ⇒ Source  50mSec  13:00:03.123
```
The source made its request and the destination responded in 2 mSec. However, it took the response 50 mSec to be received at the source. In one direction the travel time was shorter than the return. The generation of the response packet was 2 mSec. This is the definition of network latency. In this example the latency was seen in one direction. Network latency can manifest in either direction or both directions at the same time.

Connectivity Testing
The same setup done for latency testing is also used for connectivity verification. In this case the goal is to determine if a packet sent arrives at the destination, and then verify that the response packet arrives at the source. The timing of the packets is not the focus. The focus of this task is to prove that packet 0x23a1 arrived at the destination. The destination sends a response packet 0xaa35 and does it arrive at the source.

Additional Information

Refer this for packet captures in NSX environment: Troubleshooting Packet Captures for VMware NSX