Potentially poor NFS Read I/O performance with 10GbE vmnics
search cancel

Potentially poor NFS Read I/O performance with 10GbE vmnics

book

Article ID: 326262

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

In vSphere 6.0, NFS Read I/O performance (in IO/s) for large I/O sizes (of 64KB and above) with an NFS datastore may exhibit significant variations. This issue is observed when certain 10 Gigabit Ethernet (GbE) controllers are used. The performance variability reported in this KB is specific to ESXi's NFS client and does not pertain to NFS clients in a virtual machine.
 
Symptoms:
  • Varying performance (IOPS) with read workloads.
  • Physical NIC shows increasing packet error counts:
    • In vSphere 6.0
      # ethtool -S vmnicX | grep rx_errors
    • vSphere 6.5 and later
# esxcli network nic stats get -n vmnic0 | grep "Receive"


Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0

Resolution

Workaround:
 
Increase the vmnic receive ring size (rx parameter) to maximum.
  1. Open an ESXi shell. For instructions, see KB 2004746.
  2. List the vmnics: 
    esxcli network nic list
    Example output: 
    Name     PCI Device    Driver Admin Status  Link Status  Speed Duplex  MAC Address          MTU    Description
----     ----------    ------ ------------  -----------  ----- ------  -----------          ---    --------------- 
...  vmnic6   0000:24:00.0   ixgbe   Up            Up           10000  Full   xx:xx:xx:xx:xx:xx  1500  Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection
...
  3. Check the specified vmnic for receive ring parameter information:

    # esxcli network nic ring preset get -n  

    This gets the ring size the nic can support as set by the manufacturer.

    # esxcli network nic ring current get -n vmnicX

    This gets the current setting of the nic.

    Important: Do the next steps with the ESXi host in maintenance mode, to avoid any potential production impact. For more information, see the Place a Host in Maintenance Mode section of the VMware vSphere Product Documentation. 
  4. Using the Intel 82599 NIC as an example, the following step changes the rx ring parameter from 456 (default) to 4096 (max):

    # esxcli network nic ring current set -n vmnicX -r 4096

    This replace 4096 with the output of Step 3 esxcli preset command. 

    Notes: ​​​​​
  • If this still generates errors, try setting the number to something lower; for example, 2040 or 1020. Once set, these changes are written in the esx.conf file, which makes them persistent across reboot. To make sure this happens in ESXi 6.5 or 6.7, you must use ESXi 6.5 P03 or later, or ESXi 6.7 U3 or later.
  • If you update local.sh to persist the change across reboots of the ESXi host, substitute localcli for esxcli. For more information, see See Modifying the rc.local or local.sh file in ESX/ESXi to execute commands while booting (2043564).

Additional Information

For translated versions of this article, see:
Powered by Wolken Software