NSX Edge ARP using Incorrect Source IP address

Products

• VMware NSX Data Center for vSphere

Issue/Introduction

Symptoms:

• No connectivity between the edge and it's default gateway.
• Packet captures indicate that the edge is sending ARP requests using a source IP that is not on the same subnet as the requested IP address

Environment

VMware NSX Data Center for vSphere 6.4.x
VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.3.x

Cause

• Issue occurs when the uplink interface has two primary IP addresses in separate subnets.
• Can occur when traffic is configured to use the primary IP address in the different a subnet to the default gateway, for example when NAT is configured. 
• This is default behavior on the NSX Edge / Linux machines based on the following Linux kernel parameters:
  • net.ipv4.conf.all.arp_announce = 0
  • net.ipv4.conf.default.arp_announce = 0

Resolution

To force the Edge to use a Source IP in the same subnet, run the following API call:

Method: PUT
URL: https://NSX_Manager_IP/api/4.0/edges/EDGE-ID/systemcontrol/config
Body:
<systemControl>
<property>sysctl.net.ipv4.conf.all.arp_announce = 2</property>
<property>sysctl.net.ipv4.conf.default.arp_announce = 2</property>
</systemControl>

Confirm new settings by using a GET of the same call.