vMotion fails and unable to set MTU of 9000 on ESX/ESXi with some versions of nx_nic, unm_nic and bnx2 drivers

Products

• VMware vSphere ESXi

Issue/Introduction

• Under certain conditions, vMotion fails over NICs supported by some versions of nx_nic and unm_nic drivers
• You can also unable to set the MTU to 9000 on a NC375T or Broadcom NetXtreme II BCM5709 1000 Base-T NICs. While setting up the MTU you may get an error:
  
  
  #esxcfg-vswitch -m 9000 vSwitch1
  Unable to set MTU to 9000 the following uplinks refused the MTU setting:vmnic1
  
  
• In the vSphere Client, you see the error:
  
  A general system error occurred: Migration failed while copying data. Connection closed by remote host, possibly due to timeout.

Environment

VMware ESX 4.0.x
VMware ESXi 4.1.x Installable
VMware ESX Server 3.5.x
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Installable
VMware ESX 4.1.x
VMware ESXi 4.0.x Installable

Resolution

This issue occurs when you attempt a vMotion operation over NC522SFP, NC522m, NC375i, and NC375T adapters with VLAN and TSO enabled, using nx_nic driver versions 4.0.407, 4.0.550 and 4.0.560 and unm_nic driver versions 4.0.406 and 4.0.555.

If you are running Proliant server with NC375T with firmware version 4.0.555 or 4.0.556 you will not be able to set the MTU of 9000 on the NICs. Similar issue found on Broadcom NetXtreme II BCM5709 1000 Base-T using bnx2 driver version 2.1.12b.v50.3 and firmware 6.2.16 bc 5.2.3 NCSI 2.0.11.

Note: On HP servers, issue of setting up MTU of 9000 was not observed on the embedded NC375i NICs.

Resolution:

To resolve this issue of vMotion on ESX/ESXi 4.x, install VMware ESX/ESXi 4.X Driver for QLogic Intelligent Ethernet Adapters version 4.0.570.

To resolve the issue of setting up MTU of 9000 on NC375T NICs, for now downgrade the firmware to 4.0.544 as HP has released an advisory which discusses this issue and HP will release a fix with no ETA at present.

Workaround:

Use one of the workaround options appropriate for your version of ESX/ESXi.

Version	Workaround Options
ESX 4.0	Do not configure VLAN for the VMkernel interface used for vMotion. Disable TSO through the service console. Create a VMkernel interface with TSO disabled by running the command: esxcfg-vmknic -t -a -i <ip_address> -n <netmask> pgName Enable vMotion on the VMkernel interface by running the command: vimish -n - "/hostsvc/vmotion/vnic_set <vmkernel_interface>"
ESXi 4.0	Do not configure VLAN for the VMkernel interface used for vMotion. Disable TSO by using the RCLI. Create a VMkernel interface with TSO disabled by running the command: vicfg-vmknic.pl -t -a -i <ip_address> -n <netmask> pgName --server <esxi_server_ip> Enable VMotion on the VMkernel interface by running the command: vicfg-vmknic.pl -E pgName --server <esxi_server_ip>
ESX 3.5	Do not configure VLAN for the VMkernel interface used for vMotion. Disable TSO through the service console. Create a VMkernel interface with TSO disabled by running the command: esxcfg-vmknic -t -a -i <ip_address> -n <netmask> pgName Enable vMotion on the VMkernel interface by running the command: vimish -n -e "/hostsvc/motion/vnic_set <vmkernel_interface>"
ESXi 3.5	Do not configure VLAN on the VMkernel interface used for vMotion.