Potentially poor NFS Read I/O performance with 10GbE vmnics

Products

• VMware vSphere ESXi

Issue/Introduction

In vSphere 6.0, NFS Read I/O performance (in IO/s) for large I/O sizes (of 64KB and above) with an NFS datastore may exhibit significant variations. This issue is observed when certain 10 Gigabit Ethernet (GbE) controllers are used. The performance variability reported in this KB is specific to ESXi's NFS client and does not pertain to NFS clients in a virtual machine.

 

Symptoms:

• Varying performance (IOPS) with read workloads.
• Physical NIC shows increasing packet error counts:
  • In vSphere 6.0

    # ethtool -S vmnicX | grep rx_errors

  • vSphere 6.5 and later

    # esxcli network nic stats get -n vmnic0 | grep "Receive"

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0

Resolution

Workaround:

 

Increase the vmnic receive ring size (rx parameter) to maximum.

1. Open an ESXi shell. For instructions, see Using ESXi Shell in ESXi
   .
2. List the vmnics:

   esxcli network nic list

   Example output:

   Name     PCI Device    Driver Admin Status  Link Status  Speed Duplex  MAC Address          MTU    Description
   ----     ----------    ------ ------------  -----------  ----- ------  -----------          ---    --------------- 
   ...  vmnic6   0000:24:00.0   ixgbe   Up            Up           10000  Full   xx:xx:xx:xx:xx:xx  1500  Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection
   ...
    

3. Check the specified vmnic for receive ring parameter information:

   # esxcli network nic ring preset get -n  

    

   This gets the ring size the nic can support as set by the manufacturer.

   # esxcli network nic ring current get -n vmnicX

    

   This gets the current setting of the nic.
   
   Important: Do the next steps with the ESXi host in maintenance mode, to avoid any potential production impact. For more information, see the Place a Host in Maintenance Mode section of the VMware vSphere Product Documentation. 
4. Using the Intel 82599 NIC as an example, the following step changes the rx ring parameter from 456 (default) to 4096 (max):

   # esxcli network nic ring current set -n vmnicX -r 4096

   
   
   

Additional Information

Notes: ​​​​​

• If this still generates errors, try setting the number to something lower; for example, 2040 or 1020. Once set, these changes are written in the esx.conf file, which makes them persistent across reboot. To make sure this happens in ESXi 6.5 or 6.7, you must use ESXi 6.5 P03 or later, or ESXi 6.7 U3 or later.
• If you update local.sh to persist the change across reboots of the ESXi host, substitute localcli for esxcli. For more information, see See Modifying the rc.local or local.sh file in ESX/ESXi to execute commands while booting.