ESXi may lose management network connectivity or disconnect from vCenter when using a Broadcom NetXtreme-E network card

Products

• VMware vSphere ESXi

Issue/Introduction

This article describes the disruption of management network connectivity after failover or failback when using Broadcom NetXtreme-E network cards.

• After network failover or failback (vmnic link state change), ESXi management network becomes unreachable and your ESXi hosts keep dropping out of vCenter.
• On physical switches, you can see vmk0's MAC address (usually the MAC address of vmnic0, but it is possible vmk0 has the MAC of another vmnic) on two switchports.
• You may also see other NetXtreme-E network cards' MAC addresses (not only vmnic0) on the physical switch.

Cause

This family of network adapters has an embedded Link Layer Discovery Protocol (LLDP) engine that will periodically send LLDP packets with the vmnic's MAC to the physical switch.

By design, vmk0 copies the physical MAC address of vmnic0 (or the first vmnic ESXi "sees" during initial installation).

• This is useful for some customers as they can prepare DHCP reservations ahead of the hardware arriving as server vendors will give you the MAC address for the onboard adapters once it is assembled.

If vmk0 is on a portgroup using the native (untagged) VLAN, this can cause the host to disconnect from vCenter.

Example:

• vmnic0 and vmnic1 are configured as Active/Active in the portgroup's teaming/failover settings, and vmk0 is currently using vmnic0.
  • You can check to see which vmnic is assigned to the vmk from an SSH session to the host, and running esxtop then press n (for Network view).
• vmnic0 has MAC ##:##:##:aa:aa:aa and vmnic1 has MAC ##:##:##:bb:bb:bb.
• vmk0 will also have MAC ##:##:##:aa:aa:aa.
• LLDP packets are coming from vmnic0 itself with MAC ##:##:##:aa:aa:aa on the native VLAN
• vmk0's traffic is also being sourced from vmk0's MAC ##:##:##:aa:aa:aa -- also on the native VLAN.
  
  No disruption occurs in the above configuration/scenario, since the ##:##:##:aa:aa:aa MAC is only seen on this one physical switchport.

• If vmk0 then moves to vmnic1 (due to a host reboot, vmnic0 going down, or the management portgroup's teaming/failover setting being configured to use vmnic1), the physical switch will see the ##:##:##:aa:aa:aa MAC on two switchports at the same time (or flapping between the ports) on the same VLAN.

Troubleshooting: 
If you have syslog configured for your physical switches, you may see Mac Flapping (MACFLAP on Cisco) entries in the switch logs. This may also manifest itself in the switch logs as a high volume of CAM table updates as the two ports fight over the MAC Address.

Resolution

To resolve this issue, disable LLDP Nearest Bridge in the BIOS Device Configuration Menu (and disable LLDP or set it to Listen on the vDS, if configured): 

• UEFI HII Menu for Ethernet Network Adapters
• NeteXtreme-E User Guide for Dell Platforms

To work around this issue, or if you do not wish to disable LLDP, you can do any of the options below:

• Delete/recreate vmk0 so that it has a unique VMware-assigned MAC address.
  • Configuring Standard vSwitch (vSS) or virtual Distributed Switch (vDS) from the Comand Line in ESXi
• Put vmk0 on a tagged NON-Native VLAN. This will prevent conflict with the LLDP agent and MAC address if it's on a completely different broadcast domain.
• Disable the vmnic that matches vmk0's MAC.
• Set a Teaming and Failover policy that ensures vmk0 can only use the vmnic with the matching MAC.

 

Additional Information

Impact/Risks:

ESXi Management network becomes unreachable for a short period of time