VMXNET3 resource considerations on a Windows virtual machine that has vSphere DirectPath I/O with vMotion enabled
search cancel

VMXNET3 resource considerations on a Windows virtual machine that has vSphere DirectPath I/O with vMotion enabled

book

Article ID: 343839

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

In a Cisco Unified Computing System (UCS) that has Cisco Virtual Machine Fabric Extender (VM-FEX) deployed, you can configure resources and offload settings for VMXNET3 adapters in an adapter policy and apply the policy to multiple hosts. The policy must have enough queue and interrupt resources to satisfy the VMXNET3 driver configuration in the guest operating system. If you use the vSphere DirectPath I/O with vMotion passthrough technology for high networking performance, the number of transmit (Tx) queues, receive (Rx) queues, completion queues (CQs), and interrupt vectors set in the VMWarePassThru adapter policy in the UCS Manager must match the numbers that are configured for the VMXNET3 driver in the guest operating system.

For information about configuring Cisco VM-FEX on vSphere, see the Cisco UCS B-Series Servers documentation.

For best practices in configuring Cisco VM-FEX and vSphere including vSphere DirectPath I/O with vMotion, see Cisco UCS Manager VM-FEX for VMware GUI Configuration Guide

For information about resource considerations for using VMXNET3 with vSphere DirectPath I/O with vMotion on Linux virtual machines, see Managing VMXNET3 resource requirements on a Linux virtual machine that has vSphere DirectPath I/O with vMotion enabled (2058349).

Environment

VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.0
VMware vCenter Server 6.0.x
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.5

Resolution

Resource allocation for VMXNET3 networking in vSphere DirectPath I/O with vMotion

For a multiprocessor system, such as an ESXi host, increasing the number of queues increases the data that is transmitted and received in parallel. Transmitting and receiving data in parallel increases throughput and reduces latency. The packets in a particular transmit queue or receive queue can be processed by a specific virtual CPU. For improved performance you can use as many queues as the number of virtual CPUs in a virtual machine.

To send or receive packets simultaneously in multiple queues on a host, processing of each queue must be triggered by an individual interrupt line. In a system that requires many interrupts, MSI or MSI-X interrupts are used instead of wired ones. The default interrupt mode for the VMXNET3 adapter in passthrough mode is MSI-X. In passthrough mode, VMXNET3 adapters do not work with INTx interrupts.

One MSI or MSI-X interrupt vector is typically assigned to one queue. When a VMXNET3 adapter that is used for vSphere DirectPath I/O with vMotion sends or receives data, the interrupt vectors that are assigned to the adapter are allocated directly on the physical host. An ESXi host has a finite number of interrupt vectors for I/O device operations. Under certain conditions, the number of vectors might be exhausted when a great number of virtual I/O device requests are generated, for example, in workloads with high packet rates and many queues.

Determining resource requirements

On a Windows guest operating system, the default amount of resources that the VMXNET3 driver consumes depends on whether Receive Side Scaling (RSS) is enabled. By default, RSS is disabled for the VMXNET3 driver.

To check whether RSS is enabled for all virtual machine adapters on the Windows guest operating system, run the NetShell command-line tool in the guest operating system.

  1. At the command prompt, run the following console command:

    netsh int tcp show global

  2. In the command output, verify that the Receive-Side Scaling State parameter is enabled.

Allocated resources cannot be examined from within the guest operating system unless they have been manually configured. The upper bound of the resources that are used in Windows for an VMXNET3 adapter is as follows:

RSS State Resource Type Resource Amount or Mode
Disabled Number of Tx queues 1
Number of Rx queues 1
Number of CQs 2
Number of interrupt vectors 2
Interrupt mode MSI-X
Enabled Number of Tx queues

The VMXNET3 driver limits the number of Tx queues in the following way:

  1. Select the minimum value from the number of vCPUs and configured maximum number of Tx queues.
  2. Round off the selected minimum to the nearest power-of-2 number in descending direction.

The following expression represents the calculation:
RoundDownToPowerOf2(Min(Number of vCPUs, Configured max number of Tx queues))
Number of Rx queues

The VMXNET3 driver limits the number of Rx queues in the following way:

  1. Select the minimum value from the number of vCPUs and configured maximum number of Rx queues.
  2. Round off the selected minimum to the nearest power-of-2 number in descending direction.

The following expression represents the calculation:
RoundDownToPowerOf2(Min(Number of vCPUs, Configured max number of Rx queues))
Number of CQs 
Number of Tx queues + Number of Rx queues
Number of interrupt vectors Max(Number of Tx queues, Number of Rx queues)
Interrupt Mode MSI-X

Configuring resources

To configure the resources and the RSS mode in the Windows guest operating system, from the Device Manager edit the advanced settings of the VMXNET3 adapter.

  1. In the Windows Device Manager, expand Network adapters.
  2. Right-click the VMXNET3 adapter whose resource settings you want to configure and select Properties.
  3. In the adapter properties dialog box, click the Advanced tab.
  4. Select the resource configuration property and edit its value to meet the requirements of the virtualized workload.
  5. Click OK.

Certain restrictions apply to the maximum resources that can be configured to a VMXNET3 adapter.

Resource Type Configuration Restrictions
Maximum number of Tx queue Cannot exceed the number of vCPUs.
Maximum number of Rx queue Cannot exceed the number of vCPUs.
Maximum number of RSS processors Cannot exceed the number of vCPUs.
Interrupt mode and number of interrupt vectors Cannot be configured on the Windows guest operating system. The mode is always MSI-X.
The VMXNET3 driver determines the number of interrupt vectors per adapter. The number of interrupt vectors does not exceed Number of CQs + 2.

Resource requirements for VMXNET3 driver 1.2.22.0 or later

Version 1.2.22.0 of the VMXNET3 driver or later supports vSphere DirectPath I/O with vMotion for Windows guest operating systems. The queues and interrupt vectors that VMXNET3 driver 1.2.22.0requires when in passthrough mode are as follows:

Windows Guest Operating System RSS State Resource Type Requirements
  • Windows Server 2008 Service Pack 2
  • Windows Server 2008 R2
  • Windows Server 2012*
  • Windows Server 2012 R2*
 Disabled Number of Tx queues 1
Number of Rx queues 1
Number of interrupt vectors 2
Interrupt mode MSI-X
  • Windows Server 2008 Service Pack 2
  • WIndows Server 2008 R
  • Windows Server 2012*
  • Windows Server 2012 R2*
 Enabled
VMXNET3 adapter is configured with one Tx queue		 Number of Tx queues 1
Number of Rx queues

Determine the required number of Rx queues in the following way:

  1. Select the minimum value from the number of vCPUs, configured maximum number of Rx queues and the number 8.
  2. Round off the selected minimum to the nearest power-of-2 number in descending direction.

The following expression represents the calculation:
RoundDownToPowerOf2(Min(Number of configured Rx queues, Number of vCPUs, 8))
Number of interrupt vectors Number of Rx queues + 2
Interrupt mode MSI-X
  • Windows Server 2008 Service Pack 2
  • WIndows Server 2008 R2
  • Windows Server 2012*
  • Windows Server 2012 R2*
 Enabled
VMXNET3 adapter is configured with multiple Tx queue		 Number of Tx queues

Determine the required number of Tx queues in the following way:

  1. Select the minimum value from the number of vCPUs, configured maximum number of Tx queues and the number 8.
  2. Round off the selected minimum to the nearest power-of-2 number in descending direction.

The following expression represents the calculation:
RoundDownToPowerOf2(Min(Number of configured Tx queues, Number of vCPUs, 8))
Number of Rx queues

Determine the required number of Rx queues in the following way:

  1. Select the minimum value from the number of vCPUs, configured maximum number of Rx queues and the number 8.
  2. Round off the selected minimum to the nearest power-of-2 number in descending direction.

The following expression represents the calculation:
RoundDownToPowerOf2(Min(Number of configured Rx queues, Number of vCPUs, 8))
Number of interrupt vectors Number of Rx queues + 1
Interrupt mode MSI-X

 

Powered by Wolken Software