Two VMs on the same VLAN or subnet cannot communicate with each other due to duplicate MAC addresses on the VMs or VMkernel interfaces

Products

• VMware vSphere ESXi

Issue/Introduction

• Two VMs (VM1 and VM2) on the same VLAN or subnet cannot ping each other when hosted on different ESXi hosts. However, migrating VM2 to another host may allow VM1 to ping VM2.
• VM1 and VM2 are not completely isolated from the network; for example, VM1 can ping VM3, and VM2 can ping VM4.
• The VMs do not both learn each other’s MAC addresses; for instance, VM1 does not learn VM2’s MAC address, but VM2 learns VM1’s MAC address.
• Migrating either VM1 or VM2 to a different ESXi host may resolve the issue.

Environment

VMware ESXi 7.0.x

VMware ESXi 8.0.x

Cause

At the outset, VM1 and VM2 are unaware of each other’s MAC addresses within the same VLAN or subnet. When VM1 pings VM2, it sends an ARP request to a broadcast MAC address, which VM2 receives. VM2 then sends an ARP reply back to VM1, enabling both VMs to learn each other’s MAC addresses. If a duplicate VM MAC address exists in the same VLAN, switches may forward the ARP reply to the wrong location (where the duplicate MAC address is attached). This switch forwarding behavior disrupts traffic between VM1 and VM2.

Resolution

If the customer has access to all ESXi hosts and the duplicate MAC address is in the range 00:50:56:xx:xx:xx (indicating a likely VMware VM), the following command can be used on each host to identify the VM with the duplicate MAC:

# net-stats -l | grep 00:50:56:duplicate_MAC

# esxcfg-vmknic -l | grep 00:50:56:duplicate_MAC

Workaround Steps:

a. Perform a ping test between the two VMs to identify which VM does not learn the other’s MAC address (e.g., vm1 does not learn vm2’s MAC, but vm2 learns vm1’s MAC). Use:

    Windows VM: arp -a
    Linux VM: arp -an

b. Change the MAC address of the VM (e.g., vm1) that fails to learn the other VM’s MAC to a unique MAC address (e.g., 00:50:56:unique_MAC, example of unique_MAC: 12:34:56, avoid reusing this example MAC).

c. If the MAC address cannot be changed, clone the VM (e.g., vm1) and assign a unique MAC and IP address for testing purposes.

Note: The physical network is not managed by the vSphere support team. Identifying the source of a duplicate MAC address typically requires involving the customer’s network support team, which can be time-consuming. This workaround avoids network team involvement, enabling faster service recovery.

Additional Information

A relevant Knowledge Base (KB) article can be found at: "Virtual Machine MAC Address Conflicts or Duplicate MAC Address Issues When Creating a Virtual Machine."

This KB 313568 article, helps identify issues caused by stale VM entries on ESXi hosts that contribute to MAC address conflicts.