ESXi host with NVMe datastore over TCP shows latency

Products

VMware vSphere ESXi

Issue/Introduction

Captures on the Source host shows that for a single segment, there are around 12 ACKs sent by the target host
This causes more time for a file to be copied or transferred from one host to another.

No.        Time            Delta Time    		Source        		Destination            Source                    Destination       		Protocol       Identification   Destination Port               Info
23        11:31:31.954842    0.000000    <Source-Ip-Address>   	<Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    902                  62375 → 902 [ACK] Seq=1 Ack=1 Win=2862 Len=16384 TSval=3240733514 TSecr=2654423814
24311     11:31:35.650912    3.696070     <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                [TCP Window Update] 902 → 62375 [ACK] Seq=1 Ack=161366525 Win=1481 Len=0 TSval=2654424183 TSecr=3240733884
210563    11:32:07.604307    31.953395    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                [TCP Window Update] 902 → 62375 [ACK] Seq=1 Ack=1381153399 Win=1962 Len=0 TSval=2654427379 TSecr=3240737078
272743    11:32:17.881889    10.277582    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                [TCP Window Update] 902 → 62375 [ACK] Seq=1 Ack=1787923263 Win=1959 Len=0 TSval=2654428407 TSecr=3240738106
334943    11:32:28.563242    10.681353    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                902 → 62375 [ACK] Seq=1 Ack=2196601641 Win=630 Len=0 TSval=2654429475 TSecr=3240739174
397010    11:32:39.065266    10.502024    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                902 → 62375 [ACK] Seq=1 Ack=2606469013 Win=1823 Len=0 TSval=2654430525 TSecr=3240740225
458795    11:32:49.148276    10.083010    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                902 → 62375 [ACK] Seq=1 Ack=3018477161 Win=1900 Len=0 TSval=2654431533 TSecr=3240741233
582599    11:33:09.913022    20.764746    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                902 → 62375 [ACK] Seq=1 Ack=3846775637 Win=1920 Len=0 TSval=2654433610 TSecr=3240743309
644481    11:33:20.455545    10.542523    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                902 → 62375 [ACK] Seq=1 Ack=4262801709 Win=1886 Len=0 TSval=2654434664 TSecr=3240744364
706660    11:33:31.316176    10.860631    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                [TCP Window Update] 902 → 62375 [ACK] Seq=1 Ack=376505117 Win=1888 Len=0 TSval=2654435750 TSecr=3240745450
768909    11:33:41.872474    10.556298    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                902 → 62375 [ACK] Seq=1 Ack=793051857 Win=1881 Len=0 TSval=2654436806 TSecr=3240746505
830883    11:33:52.835510    10.963036    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                [TCP Window Update] 902 → 62375 [ACK] Seq=1 Ack=1200360255 Win=1871 Len=0 TSval=2654437902 TSecr=3240747601
892862    11:34:03.222765    10.387255    <Source-Ip-Address>   <Target-Ip-Address>        <Source-Mac-Address>    <Target-Mac-Address>        TCP            xxxx (xxxx)    62375                [TCP Window Update] 902 → 62375 [ACK] Seq=1 Ack=1611476683 Win=753 Len=0 TSval=2654438941 TSecr=3240748641

Environment

ESXi 7.x/8/x

Cause

TCP_DELAYED_ACK holds ACKs for small, individual data segments, causing the sender to wait for a prolonged time and thus slowing down transfers.

Resolution

NOTE: Please contact your storage vendor to find out if they recommend disabling delayed ACK to enhance performance and reduce potential issues with their storage systems.
The final conclusion to whether or not disable the TCP_DELAYED_ACK parameter should be confirmed by the customer's storage vendor.

Login to the ESXi host as user root and disable the _net_inet_tcp_delayed_ack by running the command below

vsish -e set /net/tcpip/instances/defaultTcpipStack/sysctl/_net_inet_tcp_delayed_ack 0
To make the changes persistent on the ESXi host post reboot, follow below steps
- Login to the ESXi host as user root
- Go to the location /etc/rc.local.d/
- vi local.sh
- Add the command just above the line that has exit 0
  
  vsish -e set /net/tcpip/instances/defaultTcpipStack/sysctl/_net_inet_tcp_delayed_ack 0

Additional Information

After setting the _net_inet_tcp_delayed_ack to 0, i.e disabled, the same file copy gets completed with less time.
Below is the capture shown on the source ESXi host vmnicX.
The capture after disabling TCP_DELAYED_ACK shows that the number of repetitive TCP ack is reduced.

No. Time DeltaTime Source Destination Source Destination Protocol Destination Port Info
118990 11:36:52.237420 5.651622 <Source-Ip-Address> <Destination-Ip> <Source-Mac> <Destination-Mac> TCP 902 38727 → 902 [PSH, ACK] Seq=528988537 Ack=1 Win=735 Len=5792 TSval=4101898077 TSecr=4285890712 [TCP segment of a
148646 11:36:53.391808 1.154388 <Source-Ip-Address> <Destination-Ip> <Source-Mac> <Destination-Mac> TCP 38727 902 → 38727 [ACK] Seq=1 Ack=623745723 Win=2290 Len=0 TSval=4285890828 TSecr=4101898193
212011 11:36:57.131285 3.739477 <Source-Ip-Address> <Destination-Ip> <Source-Mac> <Destination-Mac> TCP 38727 902 → 38727 [ACK] Seq=1 Ack=942309931 Win=2275 Len=0 TSval=4285891202 TSecr=4101898566
274885 11:37:00.911905 3.780620 <Source-Ip-Address> <Destination-Ip> <Source-Mac> <Destination-Mac> TCP 38727 [TCP Window Update] 902 → 38727 [ACK] Seq=1 Ack=1258134293 Win=2360 Len=0 TSval=4285891580 TSecr=4101898945
302594 11:37:02.510106 1.598201 <Source-Ip-Address> <Destination-Ip> <Source-Mac> <Destination-Mac> TCP 902 38727 → 902 [PSH, ACK] Seq=1399591551 Ack=1 Win=735 Len=13656 TSval=4101899105 TSecr=4285891739 [TCP segment of a
337350 11:37:04.580918 2.070812 <Source-Ip-Address> <Destination-Ip> <Source-Mac> <Destination-Mac> TCP 38727 [TCP Window Update] 902 → 38727 [ACK] Seq=1 Ack=1575427499 Win=2346 Len=0 TSval=4285891947 TSecr=4101899312
400645 11:37:08.220673 3.639755 <Source-Ip-Address> <Destination-Ip> <Source-Mac> <Destination-Mac> TCP 38727 [TCP Window Update] 902 → 38727 [ACK] Seq=1 Ack=1896655609 Win=2371 Len=0 TSval=4285892311 TSecr=4101899676
462981 11:37:11.818949 3.598276 <Source-Ip-Address> <Destination-Ip> <Source-Mac> <Destination-Mac> TCP 38727 902 → 38727 [ACK] Seq=1 Ack=2213915033 Win=2313 Len=0 TSval=4285892671 TSecr=4101900035

Refer to the KB Packet capture on ESXi using the pktcap-uw tool for more info on running the packet capture tool