Multihoming on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information about multihoming VMkernel networking.

Multihoming in a VMkernel networking context means that there are multiple VMkernel adapters (vmknic) in a single TCP/IP stack (Netstacks).

 

Observed issues that may be present:

• vMotion operations can fail or take an exceedingly long time.
• Storage latency may be reported (iSCSI not using port binding).
• ESXi host disconnects from vCenter can occur.
• ICMP replies to vmks may not be received.
• vSphere/vSAN Health Check failures, including "vMotion unicast connectivity check", "vMotion MTU check", or "vSAN MTU check" alarms.

The above may exist if you have VMkernel ports configured on the same IP subnet and TCP/IP stack similar to:

• Managment vmk0 with IP 192.#.#.8/24 and Default Gateway 192.#.#.1 as part of the default TCP/IP stack.
• vMotion vmk1 with IP 192.#.#.10/24 and Default Gateway 192.#.#.1 as part of the default TCP/IP stack.
• iSCSI vmk2 with IP 192.#.#.12/24 and Default Gateway 192.#.#.1 as part of the default TCP/IP stack.

In the above configuration, all of the vmknics are in the same IP subnet and TCP/IP stack. This results in the TCP/IP stack potentially selecting the wrong vmknic for egress vmknic traffic and can lead to potential issues with connectivity, low throughput, and asymmetric routing

Cause

Configurations with more than one vmknic in the same IP subnet and TCP/IP stack are not supported.

NOTE: There are exceptions:

• iSCSI port binding
• Multi-NIC vMotion
• NSX TEPs

Resolution

Ensure the VMkernel ports are in a different IP subnets or have their own TCP/IP stack.

Examples:

1. Supported VMkernel port configuration:
   1. vMotion vmk1 and IP 192.#.#.10/24 with Default Gateway 192.#.#.1/24 (in the pre-defined vMotion TCP/IP stack).
   2. iSCSI vmk2 and IP 172.#.#.12/24 with Default Gateway 172.#.#.1/24 (in a Custom TCP/IP stack).
2. Supported VMkernel port configuration example: 
   1. Management vmk0 and IP 192.#.#.10/24 with Default Gateway 192.#.#.1/24 (in the pre-defined Default TCP/IP stack).
   2. vMotion vmk1 and IP 10.#.#.12/24 with Default Gateway 192.#.#.1/24 (in the pre-defined Default TCP/IP stack).
   3. iSCSI vmk2 and IP 172.#.#.12/24 with Default Gateway Override, using gateway 172.#.#.1/24 (in the pre-defined Default TCP/IP stack).

Additional Information

• For more information about TCP/IP stacks in vSphere ESXi, see View TCP/IP Stack Configuration on a Host
• For more information and exceptions about iSCSI port binding, see Considerations for using software iSCSI port binding in ESX/ESXi
• For more information and exceptions about Multiple-NIC vMotion, see Multiple-NIC vMotion in vSphere
• For more information about VMkernel Networking in vSphere, see VMkernel Networking Layer

NOTE:

• Configurations with more than one vmknic in the same IP subnet and TCP/IP stack are not supported.
• vSAN environment also doesn't support multihoming (same IP subnet) across vmk adapters.