Send and receive buffer sizes of the TCP socket used for iSCSI connection through software iSCSI adapter are tunable on vSphere ESXi 7.0 U3d and later versions

Products

• VMware vSphere ESXi

Issue/Introduction

Starting with vSphere 7.0 U3d and later versions, Software iSCSI TCP send and receive socket buffer sizes may be modified to better match the Bandwidth Delay Product of the network being used, resulting in improved transfer throughput.  (In vSphere 7.0u3 or earlier releases, the software iSCSI socket buffer sizes are fixed.)  In general, these parameters should only be modified as directed by VMware Support.

Environment

VMware vSphere ESXi 7.0.3

Resolution

Bandwidth Delay Product (BDP) and Socket Buffer Sizes

Consider an iSCSI connection between an iSCSI Initiator (ESXi) and an iSCSI Target

• Let BW = bandwidth of network connection (unit: Gbps)
• Let RTT = round trip network latency (unit: s)
• Define Bandwidth Delay Product (BDP) = BW * RTT (BDP unit: Bytes)

For iSCSI (TCP) data transfer from Sender to Receiver (say Sender = ESXi, Receiver = iSCSI Target), the maximum transfer throughput between Sender and Receiver may be achieved when:

• Sender's send buffer size >= BDP
• Receiver's receive buffer size >= BDP

If these conditions are satisfied, a sender could send a sufficient number of unacknowledged bytes on the network to keep the network pipe full..  

As network bandwidth increases (10G, 25G, 50G, ...), the ability to increase ESXi software iSCSI send/receive buffer sizes to match the BDP of the network is necessary for achieving high data transfer throughput.
 

Bandwidth Delay Product Examples

Bandwidth	Delay  (Round Trip Time)	Bandwidth Delay Product (Bandwidth * Delay)
10Gbps = 1.25 GB/s	100us	125KB
25Gbps = 3.12 GB/s	100us	312KB
50Gbps = 6.25 GB/s	100us	625KB
10Gbps	1.5ms	1.8MB
25Gbps	1.5ms	4.6MB

 

The above examples help the reader understand that ESXi software iSCSI TCP send and receive buffer sizes could be a performance limiting factor, especially for high bandwidth or high latency networks.  The receive buffer size affects read performance, while the write buffer size affects write performance.
 

vSphere Releases and ESXi software iSCSI Socket Buffer Sizes

vSphere Release	Buffer	Default	Min	Max	Configurable?
7.0 U3 and earlier	Send	600KB			Fixed
	Receive	256KB			Fixed
7.0 U3d,8.0 and later	Send	600KB	600KB	6MB	Configurable
	Receive	256KB	256KB	6MB	Configurable

Modifying ESXi software iSCSI Socket Buffer Sizes

• 7.0 U3 and earlier
  • ESXi software iSCSI send and receive buffer sizes cannot be modified.
• 7.0 U3d, 8.0 and later
  • Existing iSCSI connections need to be re-established for new socket buffer sizes to take effect.
  • To get current buffer sizes
    • For send buffer
      • Usage: esxcli system settings advanced list -o /ISCSI/SocketSndBufLenKB
    • For receive buffer
      • Usage: esxcli system settings advanced list -o /ISCSI/SocketRcvBufLenKB
  • To set buffer sizes
    • Ensure all software iSCSI sessions are removed
      • Usage: esxcli iscsi session remove -A <software iSCSI adapter>
      • Example: esxcli iscsi session remove -A vmhba64
    • Set send buffer size
      • Usage: esxcli system settings advanced set -o /ISCSI/SocketSndBufLenKB -i <Value in KiloBytes>
      • Example: esxcli system settings advanced set -o /ISCSI/SocketSndBufLenKB -i 2048
        • Set send buffer size to 2048 KBytes
    • Set receive buffer size
      • Usage: esxcli system settings advanced set -o /ISCSI/SocketRcvBufLenKB -i <Value in KiloBytes>
      • Example: esxcli system settings advanced set -o /ISCSI/SocketRcvBufLenKB -i 3072
        • Set receive buffer size to 3072 KBytes
    • Create new software iSCSI session
      • Usage: esxcli iscsi session add -A <software iSCSI adapter>
      • Example: esxcli iscsi session add -A vmhba64