Intermittent connectivity between VMs on different ESXi hosts due to duplicate MAC addresses

Products

• VMware NSX

Issue/Introduction

VMs residing on NSX overlay segments experience intermittent network connectivity issues. Impacted VMs may fail to ping each other or fail to join the domain.

This behavior is specifically observed when the communicating VMs are located on different ESXi hosts. If both VMs are migrated via vMotion to the same host, connectivity is restored.

Symptoms include:

• ICMP request successfully leaves the source vNIC and exits the source ESXi uplink via GENEVE encapsulation.

• The destination ESXi host never receives the packets on its physical uplink.

• The NSX ARP/VTEP table may show a VM's MAC address being learned from the wrong ESXi host.

• Verification Commands: To identify these symptoms in the NSX Controller tables, run below command on the ESXi:
  
  nsxcli -c get logical-switch <VNI> arp
nsxcli -c get logical-switch <VNI> mac
nsxcli -c get logical-switch <VNI> vtep-table

Environment

VMware NSX

Cause

The issue is caused by a duplicate MAC address conflict on the overlay network. This typically occurs when VMs are cloned or restored with their vNIC MAC address settings set to "Manual".

When two active VMs share the same MAC address on the fabric, the overlay network may learn the MAC from the incorrect host, causing packets to be routed to the wrong destination host.

Resolution

To resolve this conflict and restore stable connectivity, perform the following steps:

1. Identify the impacted VMs sharing the duplicate MAC address.

2. Power down the impacted VMs.

3. Edit the VM settings and change the vNIC MAC address assignment from Manual to Automatic.

4. Power on the VMs to allow them to pull a fresh, unique MAC address from the system.

5. Verify connectivity by attempting to ping between VMs and joining the domain.