Connectivity Failure for Virtual Machines on Bridged Overlay Segments

Products

• VMware NSX

Issue/Introduction

Virtual machines connected to a bridged overlay segment are unable to communicate with external networks. Packet captures indicate that traffic from the impacted virtual machines is not reaching the NSX Edge nodes. Triage reveals a MAC address mismatch where traffic is being steered toward an incorrect destination despite using the correct logical gateway IP.

Symptoms include:

• Total loss of North-South connectivity for a specific segment.

• Packet captures showing traffic destined for the default gateway IP reaching a different virtual machine.

• "Destination MAC" in packet captures does not match the NSX Gateway/Router MAC.

Environment

VMware NSX

Cause

An IP address conflict exists where a virtual machine on the same segment is incorrectly configured with the same IP address as the default gateway. This causes ARP resolution to map the gateway IP to the virtual machine's MAC address instead of the NSX Edge/Gateway interface.

Resolution

Please follow these steps -

• Identify the Conflict:

  • Perform a packet capture on the physical switchport or the ESXi uplink and filter for traffic originating from the affected VM.
    For e.g.,
    
    The following command is capturing traffic on vmnic0.
    [root@esx:~] pktcap-uw --uplink vmnic0 --capture UplinkSndKernel,UplinkRcvKernel -o - | tcpdump-uw -enr - | grep <ip-address>
    
    Highlighted below are the destination IP and mac-address. The destination mac address should be the mac address of correct default gateway interface.
    05:10:27.588818 00:50:56:66:##.## > 00:50:56:67:##.##, ethertype 802.1Q (0x8100), length 160: vlan 1#, p 0, ethertype IPv4 (0x0800), 192.168.##.##.54830 > 192.168.##.##.6081: Geneve, Flags [none], vni 0x12000, proto TEB (0x6558), options [8 bytes]: 00:50:56:9d:##.## > 00:50:56:9d:##.##, ethertype IPv4 (0x0800), length 98: 172.16.10.## > 172.16.20.##: ICMP echo request, id 12040, seq 3, length 64

  • Compare the Destination MAC address of the frames with the MAC address of the actual Default Gateway.

• Locate the Rogue Device:

  • Identify the virtual machine currently responding to ARP requests for the gateway IP. Please refer to following KB for troubleshooting steps -
    Troubleshoot duplicate IP without shutting down known device

• Remediate the Duplicate IP:

  • Remove or change the duplicate gateway IP address from the misconfigured virtual machine.

  • Clear the ARP cache on the impacted virtual machines if connectivity does not resume immediately.

 

 

Additional Information

For advanced diagnostic steps, see Troubleshooting NSX using Packet Captures.