Enabling and verifying IOAT and Jumbo frames

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article provides information on the different ways to enable IOAT and Jumbo Frames.

Environment

VMware ESX 4.1.x
VMware ESX Server 3.5.x
VMware ESXi 4.1.x Installable
VMware ESXi 4.0.x Installable
VMware vCenter Server 5.5.x
VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 5.1
VMware vCenter Server 5.1.x
VMware ESX 4.0.x
VMware ESXi 3.5.x Installable
VMware vSphere ESXi 5.5
VMware VirtualCenter 2.5.x

Resolution

IOAT

IOAT was experimental support in ESX 3.5 and is not supported in ESXi/ESX 4.x and later. IOAT v1 is a chipset on the motherboard that needs to be enabled from hardware side.

To enable I/OAT:

Check BIOS Setting for an IOAT enabled Processor [woodcrest].
Dual-Core Intel Xeon Processor 5100 Series:
- Intel reliable memory technology called FB-DIMMs
- Intel Virtualization Technology ( Intel VT)
- Intel Active Server Manager (Intel AMT)
- Intel I/O Acceleration Technology (Intel I/OAT)
Boot the ESX host.
Change the BIOS settings:
1. Select Integrated Devices (Enter).
2. Enable IOAT DMA Engine.
3. Press ESC, save change and exit.

To enable IOAT on a VMkernel:

Run this command to check the IOAT value:

# esxcfg-advcfg --get /Net/TcpipUseIoat Value of TcpipUseIoat is 0 (off)
Run this command to enable IOAT by changing the value to 1:

# esxcfg-advcfg -s 1 /Net/TcpipUseIoat
Run the rebootcommand to reboot.

To enable IOAT using VMware Infrastructure 3 Client:

Click Configuration > Advanced Settings > Net > Net.TcpipUseIoat .
Type 1.
Reboot.

For more information, see Configuring advanced options for ESX/ESXi (1038578).

To verify if IOAT is loaded:

Run this command:

# vmkload_mod -l

The output appears similar to:

Name R/O Addr Length R/W Addr Length
ID Loaded
vmklinux 0x877000 0x1f000 0x286d700 0x4b000
vmkapimod 0x9a3000 0x4000 0x23840b0 0x2000
1 Yes acpi 0x9a7000 0x26000 0x237dc78 0x2000
2 Yes vmklinux 0x9cd000 0x29000 0x23a63c0 0x33000
3 Yes ioat 0x9f6000 0x3000 0x24024e8 0x3000
4 Yes aic79xx 0x9f9000 0x35000 0x2405838 0x9000
5 Yes
Check the VMkernel log file for any error messages.

Note: There are no documentation available, as this feature is still under experimental support.

Jumbo Frames

Jumbo Frames is officially supported in ESXi/ESX 4.x and later. It was experimental support for NAS and iSCSI traffic in ESX 3.5. For Jumbo Frames to be effective, the network must support Jumbo Frames (end-to-end). Configuration of Jumbo Frames on virtual switches is currently only available from the command line. Configuration of Jumbo Frames on vNetwork Distributed Switches can be done from vCenter.

For details on configuring Jumbo Frames on a vNetwork Distributed Switch, see Enabling Jumbo Frames on virtual distributed switches (1038827).

Notes:

In ESX 3.5, Jumbo Frames are supported on 1GB NICs and you cannot update the MTU on a pre-existing port group, this must be done on creation.
In ESX 4.x and 5.x, Jumbo Frames are supported on 1GB or 10GB NICs and you can update the MTU on a newly created port group or pre-existing port group.
Jumbo Frames up to 9KB (9000 bytes) are supported.
Like TSO, Jumbo Frames are supported in the guest operating system and in the ESX host kernel TCP/IP stack only in the context of ethernet network. They are limited to data networking only.

To set up and verify physical network switch for Jumbo Frames, consult your vendor documentation. This example sets up and verifies the CISCO 3750 I physical network switch for Jumbo Frames:

3750(config)# system mtu jumbo 9000
3750(config)# exit
3750# reload
3750#show system mtu
System MTU size is 1500 bytes
System Jumbo MTU size is 9000 bytes

To set up Jumbo Frames on VMkernel interface:

Run this command to create a new virtual switch with the command:

# esxcfg-vswitch -a vSwitch<#>

Where <#> is the vSwitch number. For example, vSwitch2.
Run this command to set an uplink to vswitch:

# esxcfg-vswitch -L <vmnic#> vSwitch<#>
Run this command to set MTU for the vswitch:

# esxcfg-vswitch -m 9000 vSwitch<#>
Run this command to add a new port group:

# esxcfg-vswitch -A <Portgroup Name> vSwitch<#>
Run this command to list your newly created vSwitch:

# esxcfg-vswitch -l

Note: To enable Jumbo frames on the vmkernel port, run this command:

# esxcfg-vmknic -a -i <ip address> -n <netmask> -m <MTU> <portgroup name>

You can run the following command if vmkernel port already exist.

# esxcfg-vmknic -m 9000 "portgroupname"

In a Windows 2003 virtual machine, provide a static IP on the 192.168.0.X network.

Ensure that your virtual machine is using the enhanced vmxnet driver.
If not, remove and re-add the network interface ensuringenhanced vmxnet is selected.

In a Windows virtual machine, check if Jumbo Frames are enabled in the guest operating system.

Log in to the guest virtual machine.
Right-click My Network Places and click Properties.
Right-click the network adapter and click Properties.
Click Configure.
Click the Advanced tab and change the MTU value.
Click OK to apply the changes.

To test Jumbo Frames:

Test ping to your neighbor’s virtual machine interface with the command:
- From Windows: ping -f -l 8972 a.b.c.d
- From Unix: ping -s 8972 a.b.c.d
The output appears similar to:

PING a.b.c.d (a.b.c.d) 8972(9000) bytes of data.
8980 bytes from a.b.c.d: icmp_seq=1 ttl=128 time=3.36 ms

For more information, see iSCSI and Jumbo Frames configuration on ESX/ESXi (1007654).

To test the network connectivity with Jumbo Frames, from ESX server console/SSH connection to ESX use the command:

# vmkping -s 8972 <IPADDRESS of iSCSI SAN>

You can use vmkpingcommand with-d switch to specify the do not defragment option.

If you see the error sendto failed, message too long while testing the vmkernel with MTU of 9000, see: Testing VMkernel network connectivity with the vmkping command (1003728).