Intermittent VM Network Connectivity Loss due to Physical Switch "MAC Address Table" Overflow

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machines (VMs) become unreachable over the network
• This behavior often appears random, affecting some VMs while others on the same host and port group remain functional.
• Packets from the VM successfully leave the ESXi host, but the physical switch is unable to forward the return traffic back on the physical port.
• In these scenarios, the issue typically resides in the physical network layer rather than the virtual layer.

Environment

VMware ESXi

Cause

• The issue happens due to the "MAC AddressTable" on the physical switch being full or overflown.
• When a switch's MAC table reaches its threshold capacity to maintain and add MAC Addresses, it can no longer map new or existing MAC addresses to specific ports.
• This leads to "Unicast Flooding" or dropped packets.

Resolution

• Login to the ESXi host as user root and verify the packet leaving the virtual switch port of the VM and the corresponding vmnic to which the VM is mapped to.

  • pktcap-uw --switchport <switchport-of-vm> --dir 2 -o - | tcpdump-uw -enr -
  • pktcap-uw --uplink <uplink-name> --dir 2 -o - | tcpdump-uw -enr -

• Refer to the KB 341568 to know more about using pktcap-uw on the ESXi host.

• To identify which vmnic is the VM mapped to, run the command esxtop --> Press n key. Here you will see the VM name with the specific ethernet port that is mapped to the corresponding vmnic.

• If you see the packets going out of the virtual switch port and the vmnic adapter, then coordinate with the physical network team to check the MAC table count on the physical switch.

• Additionally, the network connectivity can be tested further by "NIC toggle" (failover test) if the Port Group is configured with more than one NIC adapter.