High Latency verification on IvS
Article ID: 391204
Updated On:
Products
Issue/Introduction
Industrial vSwitch (IvS) have a capability to support realtime application traffic that requires low latency forwarding.
If there is higher latency than usual, the application could have issue and stop working. So it is importatnt to monitor the latency in the IvS.
This article describes how to know the latency metrics from IvS.
Environment
4.2.x
Resolution
If you suspect the latency is high, you can get the latency metrics by following the steps below.
With grep “PortID|maxLatency” you can get the summary of the Max Latency per Port ID.
- Get the MAC address of the NIC that is connected Profinet network
- Login to the ESXi Host via SSH
- Run the command and find the DvsPortset name for the MAC address
net-stats -l- Run the command and find Switch ID (swID) for the DvsPortset
nsxcli -c get ens switch list - Run the command and show the summary of the max latency metrics per PortID
nsxcli -c get ens latency system dump <swID> | grep -E "PortID|maxLatency"Here is the example of the command output.
[root@localhost:~] nsxcli -c get ens latency system dump 0 | grep -E "PortID|maxLatency"
Fri Feb 21 2025 UTC 20:04:05.554
maxLatency 312 354 138 283
PortID: XXXXXX74 SampleRate: 1 enableE2E: 1
maxLatency 32 267 563 229
PortID: XXXXXX79 SampleRate: 1 enableE2E: 1
maxLatency 69 352 636 269
PortID: XXXXXX82 SampleRate: 1 enableE2E: 1
maxLatency 271 58 103 31
PortID: XXXXXX85 SampleRate: 1 enableE2E: 1
maxLatency 33 211 217 26
PortID: XXXXXX88 SampleRate: 1 enableE2E: 1
maxLatency 120 58 625 314
PortID: XXXXXX91 SampleRate: 1 enableE2E: 1
maxLatency 56 284 1371502117788 21
PortID: XXXXXX94 SampleRate: 1 enableE2E: 1
maxLatency 154 61 87 32
PortID: XXXXXX97 SampleRate: 1 enableE2E: 1
maxLatency 33 49 1387551773108 26582If you want to look into the details of the specific port ID, run the command with the PortID. It can show the histogram of the latency.
The following is the example output for the PortID 67108974
[root@localhost:~] nsxcli -c get ens latency system dump 0 | grep -A17 XXXXXX74
Fri Feb 21 2025 UTC 20:12:03.788
PortID: XXXXXX74 SampleRate: 1 enableE2E: 1
Latency txLatencyHisto rxLatencyHisto vmRxLatencyHisto intrLatencyHisto
TotalSamples 376529121 329794089 329793033 697213551
minLatency 0 1 2 0
maxLatency 32 267 563 229
mean 2 2 7 1
32us 376529120 329794057 329790002 697213550
64us 1 30 3001 0
96us 0 0 5 0
128us 0 0 4 0
160us 0 0 3 0
192us 0 0 5 0
256us 0 1 7 1
512us 0 1 5 0
1024us 0 0 1 0
2048us 0 0 0 0
4096us 0 0 0 0
MAX 0 0 0 0
High latency is caused by various reasons. You should check the VM and ESXi host has the all required settings described in the IvS deployment guide.
There are some configurations required within the VM to enable IvS realtime feature.
Make sure the following attributes are configured in vmx file to enable strict latency.
ethernet1.realtime = "1"
virtualHW.version = "21"
vmxnet3.rev.90 = "TRUE"
Latency Sensitivity High is also required. Make sure it is set to High in vCenter.
Also, make sure NUMA alignment. All the VMs must have NUMA alignment with NICs and NSX configuration.
Feedback
