VMware ESXi network teaming policies

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on network teaming policies in ESXi.

Environment

VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.5
VMware ESXi 4.1.x Installable
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Installable

Resolution

Note: For NIC teaming, all physical switch ports in the same team must be in the same Layer 2 broadcast domain.

Route based on the originating portID (lb_srcid)

Select an uplink based on the portID of the virtual machine on the vNetwork Standard vSwitch (vSS) or vNetwork Distributed Switch (vDS).

All virtual machines powered-on on an ESXi host have an associated portID.

To calculate the uplink, the virtual machine uses a Modulus operation (%) with the port ID number and the number of uplinks on the vSS/vDS to create the hash. The remainder of the operation determines which uplink the virtual machine will use.

The portID assigned to a virtual machine is fixed while the VM is running on that host. Therefore, the same uplink is used. The calculation to create the hash only needs to be done once as a result.

Advantages

• Simple.
• Low resource overhead as the calculation to select an uplink only needs to be performed once.
• No changes are needed on the physical switch to implement this policy.
  

Disadvantages

• Bandwidth available to the virtual machine is limited to the speed of the uplink on the vSS or vDS.
• Is not aware of the current load on the uplinks thus there is the possibility for uplinks to be overloaded.
  

Notes:

• This is the default network teaming policy used on vSS and vDS.
• If you power off or power on a VM, it may get a new portID and the VM may move to a new uplink.
• Increasing or reducing the amount of uplinks on a vSS or vDS results in a recalculation and the virtual machines may move to a new uplink.
  

Load-Based Teaming

Load-Based Teaming (LBT) is an extension of portID where the current load on the uplinks are checked and steps will be taken to reduce the load on overloaded uplinks.

The uplinks are tested at a set interval (default is 30 seconds) and if the load exceeds a certain threshold (default above 75% usage), the portID of the highest I/O virtual machine is moved to a different uplink.

The net-lbt service is responsible for monitoring the uplinks and both the check time interval and load % can be adjusted to suit environmental needs. For more information on Load-Based Teaming and adjusting the parameters, see Load-Based Teaming in VMware ESX 4.x and VMware ESXi 4.x (1022590).

Advantages

• Simple.
• Low resource overhead as the calculation to select an uplink only needs to be performed once and check interval has minimal impact.
• No changes are needed on the physical switch to implement this policy as a port may move at most 30 seconds (or what ever the check interval is set to).
• Is aware of the current load on the uplinks and will take remediation steps to reduce the load on the uplinks.
  

Disadvantages

• Bandwidth available to the virtual machine is limited to the speed of the uplink on the vDS.
  

Note: Load based teaming is a feature of a vNetwork Distributed Switch (vDS), it is not available on a vNetwork Standard Switch (vSS)

Route based on source MAC hash (lb_srcmac)

Select an uplink based on the MAC address of the virtual machine located on the vNetwork Standard vSwitch (vSS) or vNetwork Distributed Switch (vDS).

To calculate the uplink, the virtual machine uses a Modulus operation (%) with the last two hexadecimal digits (least significant byte) of the virtual machine MAC address and the number of uplinks on the vSS/vDS to create the hash. The remainder of the operation determines which uplink the virtual machine will use.

Advantages

• Virtual machines will use the same uplink as the MAC address is static, powering on or off the virtual machine will have no impact on the uplink used.
• No changes are needed on the physical switch to implement this policy.
  

Disadvantages

• Bandwidth available to the virtual machine is limited to the speed of the uplink on the vSS or vDS.
• Is not aware of the current load on the uplinks, so there is the possibility for uplinks to be overloaded.
• Higher resource overhead then portID as the calculation to select an uplink needs to be done for every packet.
  

Note: Increasing or reducing the amount of uplinks on a vSS or vDS results in a recalculation and the virtual machines may move to a new uplink.

Route based on IP hash (lp_ip)

Select an uplink based both the source and destination IP addresses of each packet. This policy is required when using Link Aggregation policies such as Etherchannel (Cisco) or Link Aggregation Control Protocol (LACP)

To calculate the uplink, the last octet of both source and destination IPs in the packet are put through a Boolean XOR operation and it results through Modulus operation (%) to create the hash. The remainder of the operation will dictate which uplink the packet will use. When a virtual machine is communicating with multiple destination IPs, a different hash can be generated for each destination IP. Therefore, the packets can use different uplinks on the virtual switch resulting in higher potential throughput.

When using Link Aggregation the physical network switch must be configured to support this configuration. The switch detects packets from the same virtual machine on multiple uplinks which it may drop. Creating a Port channel or Link Aggregation Group (LAG) on the network switch results in the switch treating all the uplinks as a single logical link. Therefore, seeing the same virtual machine MAC address on multiple uplinks is not an issue.

Notes:

• The physical network switch must use the same algorithm for hashing as the ESXi host.
• Only online ports (link up) which are part of the Port Channel or LAG configurations are included. If a port is disconnected or brought offline (link down), then it is excluded from the configuration by both the host and physical network switch until a link up is detected again.

Advantages

• Virtual machine can access multiple uplinks when communicating with multiple destination IPs, thus the VM has potential access to bandwidth of all uplinks in the vSS or vDS.
  

Disadvantages

• Complex and difficult to troubleshoot
• Changes need to be made on the physical network switch to support this teaming policy.
• Is not aware of the current load on the uplinks thus there is the possibility for uplinks to be overloaded.
• Highest resource overhead as two calculations (XOR and Modulus) need to be performed for every packet.
• No benefit if there is only a single IP conversation.
  

Notes:

• The network failover detection policy Beacon Probing is not supported by IP hash (iphash).
• Ether channel is supported on both a vSS and vDS. LACP is only supported on vDS on ESXi 5.1 and later.

Use explicit failover order (fo_explicit)

Always use the highest order uplink (the network card which has been up the longest) from the list of Active adapters that passes failover detection criteria.

Uplinks in Active Adapters are considered first, if no adapters are usable then uplinks in Standby Adapters are used next. Uplinks in the Unused Adapters are never considered for failover.

There is no form of load balancing with this policy as only a single uplink is used.

Additional Information

How the Modulus operation calculation works to create the hash?

Modulus calculation is: <port-id> % <num-of-uplinks-on-vswitch>

Examples (with 2 uplinks on a vSS or vDS):

PortID 0 - 0%2 = Remainder 0 <== Use uplink 0

PortID 1 - 1%2 = Remainder 1 <== Use uplink 1

PortID 2 - 2%2 = Remainder 0 <== Use uplink 0

PortID 3 - 3%2 = Remainder 1 <== Use uplink 1

PortID 25 - 25%2 = Remainder 1 <== Use uplink 1

PortID 36 - 36%2 = Remainder 0 <== Use uplink 0

Load Based Teaming in VMware ESX 4.x and VMware ESXi 4.x