Multiple-NIC vMotion in vSphere

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on performing a vMotion across multiple NICs.

Resolution

To set up Multi-NIC vMotion in vSphere on a Standard vSwitch:

1. Log into the vSphere Client and select the host from the inventory panel.
2. Click the Configuration tab and select Networking.
3. Click Add Networking and choose VMkernel as the Connection Type.
4. Click Next.
5. Add two or more NICs to the required standard switch. Create a new vSphere standard switch or use an existing vSwitch.
   
   Note: Ensure that all adapters are on the same network when connected to a physical Ethernet environment. If the adapters are split across switches, then all adapters must be able to communicate across those switches.
    
6. Name the VMkernel portgroup (for example, vMotion-01), and assign a VLAN ID as required.
7. Click Use this port group for vMotion, then click Next.
8. Configure the IP address and subnet mask, then click Next..
9. Click the Properties tab of the vSwitch, select the vMotion-01 portgroup, and click Edit.
10. Click the NIC Teaming tab.
11. Under Failover Order, select Override switch failover order.
12. Configure the first adapter (for example, vmnic1) as active and move the second adapter (for example, vmnic3) to standby.
13. Click OK.
14. Under the vSwitch Properties, click Add to create a second VMkernel portgroup.
15. Name the VMkernel portgroup (for example, vMotion-02), and assign a VLAN ID as required.
    
    Note: Ensure that the both VMkernel interfaces participating in the vMotion have the IP address from the same IP subnet.
     
16. Click Use this port group for vMotion, then click Next.
17. Configure the IP address and subnet mask, then click Next.
18. Click the Properties tab of the vSwitch, select the vMotion-02 portgroup, and click Edit.
19. Click the NIC Teaming tab.
20. Under Failover Order, select Override switch failover order.
21. Configure the second adapter (for example, vmnic3) as active and move the first adapter (for example, vmnic1) to standby.
22. Click the Properties tab of the vSwitch. Select each vMotion portgroup in turn and confirm that the active and standby adapters are the reverse of each other.

 

Setting up Multi-NIC vMotion in vSphere on a Distributed vSwitch:

1. Log into the vSphere Client and click the Networking inventory.
2. Click New vSphere Distributed Switch and choose latest version
3. Name the Distributed switch (for example, Multi-NIC-vMotion).
4. Assign two or more uplink ports to the switch, then click Next.
5. Select physical adapters to each of the hosts, then click Next and Finish.
   
   Note: Ensure that all the adapters are on the same network when connected to a physical Ethernet environment. If the adapters are split across switches, then all adapters must be able to communicate across those switches.
    
6. Expand the Distributed switch you just created, click the dvPortGroup and click Edit Settings.
7. Name the dvPortgroup (for example, vMotion-01).
8. Click VLAN and assign a VLAN ID as required.
9. Click the Teaming and Failover tab, configure dvUplink1 as Active Uplink and move dvUplink2 to Standby Uplink.
10. Right-click the Distributed vSwitch, then click New Port Group.
11. Name the dvPortgroup (for example, vMotion-02).
12. Click VLAN and assign a VLAN ID as required, then click Next and Finish.
13. Select the second portgroup created, then click the Teaming and Failover tab.
14. Configure dvUplink2 as Active Uplink and move dvUplink1 to Standby Uplink.
15. Click the Hosts and Clusters Inventory tab, select a host's Networking, and click vSphere Distributed Switch.
16. Click Manage Virtual Adapters and then click Add.
17. Select the New virtual adapter option.
18. Choose VMkernel as the Virtual Adapter Type.
19. Select the vMotion-01 portgroup, click Use this port group for vMotion, then click Next.
20. Configure the IP address and subnet mask, then click Next and Finish.
21. Add another virtual adapter, then select the vMotion-02 portgroup.
22. On the Distributed vSwitch, select each dvportgroup on VMKernel Port vmk1 and vmk2 in turn, and confirm that the active and standby uplinks are the reverse of each other.
    
    Note: Ensure that the both VMkernel interfaces participating in the vMotion have the IP address from the same IP subnet.

After making these configuration changes, when you initiate a vMotion, multiple NIC ports are used. Even when performing a vMotion on just one virtual machine, both links are used.

If you don't have dedicated links for vMotion, consider using Network I/O Control. vMotion can saturate a link. When you set up Network I/O Control, and assigned the correct amount of shares, each type of traffic gets what it has been assigned.

Note: vMotion and IP-based storage traffic should not be routed, as this causes latency issues. Any internal/private subnet can work as long as it is unique and dedicated exclusively to that specific type of traffic. Routed IP storage is not supported. Follow the recommendations for IP-based storage configuration published by VMware.

Note: In a multi-NIC vMotion configuration, one VMkernel port group may be present on a standard switch and the other one on a distributed switch.

Additional Information

It's not supported to utilize the same VMkernel adapters for Multi-NIC vMotion as you use for iSCSI with Port Binding, as the Teaming and Failover options needed for each are contradictory.  See Considerations for using software iSCSI port binding in ESXi for details.