Loss of VM Network Connectivity when Port Security, MAC Address Authentication (MAB) or Device-Tracking is Configured on the Physical Switch

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

If you are using a MAC Address/Layer 2 security mechanism like port security or MAC Authentication Bypass (e.g, MAB) or SISF-based tracking on ESXi DVS/vSS Uplink ports connected to the physical switch, you may experience these symptoms:

After vMotion, virtual machine loses network connectivity
When teaming network adapters and failing one of them, a virtual machine or the ESXi host loses network connectivity
A limited number of TCP/IP connections can be established
A virtual machine cannot ping any other host on the physical network
A virtual machine cannot ping the gateway IP address
If a virtual machine is restarted, it loses network connectivity until the NIC is disabled and re-enabled
This issue can be experienced on Cisco physical switches (e.g., Catalyst 6500, Nexus 9300 series) with port security enabled,
The issue is frequently reported on ESXi 8.0 U3 using VMXNET3 adapters in environments managed via Cisco Identity Services Engine (ISE).

Notes:

Virtual machines can ping each other on same ESXi network
The virtual machine can ping its own IP address if the virtual NIC is configured with a static IP address
When the network connection is disabled and enabled inside the virtual machine, the network connection is restored, and the virtual machine can ping other machines on the network

Environment

VMware vSphere ESXi 7.x and 8.x
VMware vCenter Server 7.x and 8.x
Cisco Catalyst Physical Switches
Cisco Identity Services Engine (ISE)
Cisco Switch Integrated Security Features (SISF)

Cause

Port security and/or MAC Address based security mechanisms (e.g., MAB) restrict the input to an interface by limiting and identifying MAC addresses of the virtual machines/devices that are allowed to access the port. When a secure MAC addresses is assigned to a secure port, the port does not forward packets with source addresses outside the group of defined addresses. When a VM moves to a new physical port (via vMotion), the new port may block traffic if the VM's MAC address is not yet authorized or if the "switchport port-security maximum" has been exceeded. In ISE environments, if the required trunk port configurations are removed or not synchronized, the RARP/GARP update sent by ESXi during migration is dropped.

If port security, MAB, or a layer 2 MAC address security mechanism is enabled on the switch, the command show mac-address-table shows the virtual network adapters as having static MAC entries. When the virtual machine proceeds to connect through a different port (for example, after vMotion or a network adapter failover), its traffic is blocked on the new port. Network connection issues may occur if a switch port does not allow traffic from multiple MAC addresses.

Resolution

For more information, see Configuring Port Security in the Cisco Catalyst 6500 Release 12.2SXH and Later Software Configuration Guide.

Disable port security.
Configure port security with proper port numbers. This option provides some security.
Configure a secure static MAC address. This is the most secure option.

Disabling port security

Caution: This option does not provide any security.

To disable port security on the Cisco switch interface, run this command at Cisco switch port:

no switchport port-security

Configuring port security with proper port numbers

Run this command at Cisco switch port to set a maximum number of secure MAC addresses for the interface:

Switch(config-if) # switchport port-security maximum value

where value is the maximum number of MAC addresses

Note: The default maximum value is 1. Enter a value from 1 to 1024. Ensure that you enter a maximum value that allows for the number of virtual network adapters on the ESX host.

Configuring a secure static MAC address

To configure a secure static MAC address, run this command at Cisco switch port:

Router(config-if)# switchport port-security mac-address [sticky] mac_address [vlan vlan_ID]

where mac_address is the MAC address that you want to configure as static and vlan_ID is the VLAN in which the MAC address resides

To delete a static MAC address:

Run the command:

Router(config-if)# no switchport port-security mac-address [sticky] mac_address

where mac_address is the MAC address that you want to delete
After removing the offending MAC address the switch port link goes down. Run this command to enable the switch port:

Switch(config-if) # no shut

For more information on configuring Cisco infrastructure when Using ISE to secure ESXi hosts DVS/vSS Uplink ports , see Cisco ISE Secure Wired Access Prescriptive Deployment Guide

For more information on configuring Cisco infrastructure when Using SISF, see Configuring Switch Integrated Security Features

Cisco Switchport configurarion example

To configure MAB to allow multiple MAC addresses on an Interface connected to a ESXi host DVS/vSS port security on a Cisco switch interface:

interface <Switchport_Name>
switchport mode access
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication host-mode multi-auth <--------
mab
dot1x pae authenticator

Additional Information

To troubleshoot the issue you can perform packet captures to check for ARP packet propogation to other ESXi hosts. On the ESXi host hosting the VM with connectivity loss, an ARP request sent from the VM MAC address but the request is never received on other ESXi Hosts.

For packet capture troubleshooting, see Packet capture on ESXi using the pktcap-uw tool

To speak with a customer representative or a Support Engineer see Contact Support - Broadcom Software