ESX Host Network isolation during management VMKernel migration to VDS with LACP

Products

• VMware vSphere ESXi

Issue/Introduction

• Moving management VMKernel to LAG (Link Aggregation Group) fails and rolls back due to connectivity loss to vCenter
• Reason for rollback:: "Cannot connect to the specified gateway."
• Connectivity to vCenter, other ESXi hosts, and the default gateway is lost immediately upon enabling LACP on the physical switch ports.

Environment

• VMware ESX
• vSphere Distributed Switch (VDS)
• Arista Physical Switches
• LACP (Active/Passive) with MLAG

Cause

The physical environment must be configured to present an LACP channel to the ESX hosts. It is expected that while the LAG is created in vCenter as a single group for all hosts, the physical infrastructure MUST create a channel per host.

Each host requires its own physical LAG configured:

• Physical Channel Group 1 for host 1 that includes vmnicX and vmnicY in physical environment configuration.
• Physical Channel Group 2 for host 2 that includes vmnicX and vmnicY in physical environment configuration.
• Physical Channel Group 3 for host 3 that includes vmnicX and vmnicY in physical environment configuration.

Resolution

1. Create a Link Aggregation Group that includes the necessary vmnics (i.e. vmnic2 and vmnic3).  
2. On the physical environment, create a channel group that includes the ports for vmnic2 and vmnic3 per host.
   1. (i.e. for 2 hosts this means 2 channel groups configured in the physical switches - one for vmnic2 and vmnic3 on host 1, one for vmnic2 and vmnic3 on host 2)
3. The native VLAN has to be configured in the physical to accommodate the added VLAN tagging that may happen in the physical. LACP packets, on Arista switches, are automatically dropped if they enter the switch with a VLAN tag.
   1. See Arista documentation for Troubleshooting LACP for details, item 4.
   2. In this same document it details the necessary config to ensure the VLAN tagged packets are not dropped.

NOTE: ESX does not send LACP packets with any tagged VLAN.

Additional Information

LACP Support on a vSphere Distributed Switch

Arista documentation for Troubleshooting LACP