VMs Experiences Packet Drops After Migration from Old Edge Cluster to New Edge Cluster

Products

• VMware NSX

Issue/Introduction

After migrating workloads from a legacy NSX Edge Cluster to a new NSX Edge Cluster, the migrated Virtual Machines (VMs) lose North-South (internet) connectivity.

 

• Migrated VMs are unable to communicate with external IP addresses (e.g., 8.8.4.4).
• Packet captures taken at the ESXi host level reveal asymmetric routing:
  • Egress: Outbound ICMP echo requests from the VM successfully leave the host via the new Edge node's MAC address.
  • Ingress: Inbound ICMP echo replies from the upstream network are misdirected to the MAC address of the old Edge node.
• The old Edge node drops the return traffic and generates an "ICMP host unreachable" message back to the external source, as it no longer hosts the migrated VM.

 

Environment

VMware NSX

Cause

This issue occurs due to a Layer 2 ARP conflict caused by a configuration overlap within NSX.

Specifically, two separate segments are configured with the identical Gateway IP address ; one segment pointing to the legacy Edge node and another pointing to the new Edge node.

Because both Edges reside on the network and claim the same Gateway IP, the upstream physical router retains a stale ARP entry mapping the Gateway IP to the legacy Edge's MAC address. When return traffic enters the environment, the physical network switches it to the old Edge instead of the new Edge, resulting in a routing blackhole.

Resolution

To resolve this issue, the duplicate Gateway IP must be removed from the environment to eliminate the ARP conflict.

Steps to resolve:

1. Migrate any remaining workloads from the legacy segment (on the old Edge Cluster) to the new segment (on the new Edge Cluster).

2. Administratively disable or completely delete the overlapping segment on the legacy Edge to remove the duplicate IP configuration.

3. (Optional but recommended) Clear the ARP cache on the upstream physical router to force it to broadcast and learn the correct MAC address of the new Edge node immediately.

OR

Option 2: Re-IP the Legacy Segment (Workaround) If the legacy segment must remain active on the old Edge Cluster for other workloads:

1. Re-IP the gateway of the legacy segment to a new, non-overlapping IP address. This removes the IP conflict while keeping the segment operational for any remaining legacy workloads.

Additional Information

VMs on NSX Overlay Segment are unable to access Internet