Changing the default VMkernel gateway interface for a network when there are multiple VMkernel ports on the same subnet

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• A non-preferred VMkernel port is being used to access the default gateway.
• The esxcfg-route -l command shows the wrong VMkernel interface associated with the default gateway.

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1

Cause

When multiple VMkernel interfaces on an ESXi host exist in the same network subnet, the host will select only one VMkernel interface to use for communication within this network. This selection process is based on which VMkernel port was created first and configured with an IP address within the subnet. Any subsequently created VMkernel ports in this subnet will not be added to the host's routing table.

In some cases, this may not be desirable and could potentially lead to unexpected behavior from the host.

Note: It is best practice to have only one VMkernel port configured for each network or VLAN. The exception to this would be VMkernel ports used for iSCSI multipathing or multi-NIC vMotion. For more information, see the vSphere Networking Guide.

Resolution

To force a different VMkernel port to be used in the host's routing table, all VMkernel ports except the preferred one in the network must be deleted and then re-created:

Warning: Performing the steps below may cause loss of network connectivity to the host's management interface. VMware strongly recommends performing these steps from the direct console command line interface (Local Tech Support Mode) of the host, or by using an iLO/DRAC/IPMI console connection. For more information, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).

1. Place the host in question into maintenance mode.
2. Verify that an non-preferred VMkernel port interface is being used in conjunction with a default gateway via the ESXi Shell, or an SSH connection to the host in question:
   
   ~ #esxcfg-route -l

VMkernel Routes:
Network Netmask Gateway Interface
192.168.85.0 255.255.255.0 Local Subnet vmk0
default 0.0.0.0 192.168.85.2 vmk0
   
   In this example, VMkernel port vmk0 is currently being used for traffic between the gateway and the host.
   
   
3. Identify all VMkernel ports within the same subnet and record all settings: IP address, subnet, VLAN, MTU, vMotion, Fault Tolerance Logging, Management Traffic, and any security policies in place.
4. Using the vSphere Client, remove all VMkernel ports within the same subnet, leaving only the preferred interface.
   
   
   1. Using the vSphere Client, select the affected vSwitch and click Properties.
   2. Click the VMkernel port in question and click Remove.
   
   Note: This can also be done from the command line. For more information, see Configuring vSwitch or vNetwork Distributed Switch from the command line in ESXi/ESX (1008127).
   
   
5. Verify that the preferred VMkernel interface is now being used in conjunction with the default gateway using the esxcfg-route command via the ESXi Shell or SSH session:
   
   ~ # esxcfg-route -l

VMkernel Routes:
Network Netmask Gateway Interface
192.168.85.0 255.255.255.0 Local Subnet vmk1
default 0.0.0.0 192.168.85.2 vmk1
   
   
6. Re-create all of the remaining VMkernel ports removed in step 4 with the settings recorded in step 3.

Note: This change does not persist across reboots. When you reboot the host, the lowest numbered VMK in each subnet becomes the preferred interface again. You must either reapply the change or create the preferred VMK first (lowest numbered) among all the VMKernels in the same subnet. In the example above, vmk0 becomes the primary gateway for 192.168.85.x subnet after host reboot.

Additional Information

Configuring vSwitch or vNetwork Distributed Switch from the command line in ESXi/ESX
Using ESXi Shell in ESXi 5.x and 6.x
同じサブネット上に複数の VMkernel ポートがある場合にネットワークのデフォルトの VMkernel ゲートウェイ インターフェイスを変更する
同一个子网中有多个 VMkernel 端口时更改默认 VMkernel 网关接口