Intermittent Layer 2 and Layer 3 Reachability Issues for Cloned Virtual Machines on NSX Overlay Segments

Products

• VMware NSX

Issue/Introduction

• A newly cloned virtual machine on an NSX overlay segment experiences Layer 2 and Layer 3 reachability failures and dropped packets.

• Disconnecting and reconnecting the vNIC temporarily restores network connectivity.

• NSX Edge datapathd logs in /var/log/syslog indicate ARP resolution failures for the parent virtual machine's IP address:
  

  NSX 4119 SWITCHING [nsx@###### comp="nsx-edge" subcomp="datapathd" s2comp="neigh" tname="dp-learning3" level="INFO"] entry(######, ######) state incomp -> failed

• Entries in the NSX Controller logs in /var/log/syslog on the NSX Manager appliance, indicate IP discovery anomalies where the parent virtual machine's IP and MAC address are snooped but rejected because they do not match the authorized vCenter bindings:

  NSX 3704 - [nsx@###### comp="nsx-controller" level="INFO" subcomp="ip-discovery"] Discovered binding ip_address {#012 ip_address {#012 ipv4: #######012 }#012 prefix_length: 32#012}#012mac_address {#012 mac: #######012}#012type: ADDRESS_BINDING_TYPE_ARP_SNOOPING#012binding_time: #######012 not found in realized list for lspId ######; added

• Entries in /var/log/hostd.log on the ESXi host show continuous failed attempts by the NSX Manager to delete the logical port:
  

  Hostd[######]: [Originator@###### sub=Hostsvc.NetworkProvider opID=###### sid=###### user=nsx-user] Error deleting dvport ###### : Unable to delete DVPort “######" that is in use, use list: ######.eth0

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

When you clone a virtual machine, the guest operating system may internally cache the parent virtual machine's MAC and IP addresses in its persistent network configuration files.

If a Segment Security Profile with SpoofGuard enabled is applied to the segment, the ESXi host datapath enforces strict port bindings based on the newly generated vCenter MAC and assigned IP address. When the cloned virtual machine transmits traffic using the cached, unauthorized legacy MAC and IP address, the NSX SpoofGuard profile detects the discrepancy and drops the packets. This blocks the logical port realization and triggers synchronization churn between vCenter and NSX.

Resolution

This is a condition that may occur in a VMware NSX environment.

 

To resolve this issue, clear the legacy parent virtual machine MAC and IP addresses from the guest operating system's persistent network configuration.

1. Access the cloned virtual machine's guest operating system console.

2. Remove any stale persistent network rules (such as udev rules) referencing the parent virtual machine's MAC address.

3. Update the network configuration scripts to reflect the new vCenter-assigned MAC address and the correct IP address. Alternatively, configure the interface to inherit the hardware MAC address dynamically.

4. Perform a graceful reboot of the virtual machine to apply the network state changes.

Additional Information

NSX Administration Guide (SpoofGuard)

Understanding SpoofGuard Segment Profile