ARP proxy entries fail to realize on transport nodes

Products

• VMware NSX

Issue/Introduction

• ARP proxy entries added via Load Balancer or AVI configurations may fail to realise and function on a host.
• A large number of virtual services or Load Balancer VIPs are present, and the segment size exceeds a /24 subnet (e.g., /22 or /21), a configuration common in Azure VMware Solution (AVS) environments.
• An NSX Traceflow from the affected virtual machine to the VIP shows the packet being dropped by the Tier-1 Distributed Router (DR) with the error: "Dropped due to ARP failure".
• The distributed router instance on a host may not respond to ARP queries, potentially disrupting traffic flows that rely on ARP proxy (e.g., NSX Load Balancers and AVI Virtual Servers).
• Proxy ARP entries appear in the nest-db level of the host configuration when checked using the following command:
  

  /opt/vmware/nsx-nestdb/bin/nestdb-cli --cmd "get vmware.nsx.nestdb.ArpTableConfigMsg" --json --beautify

• However, they are not present when checked using net-vdr commands for realized ARP proxies on the host:
  

  net-vdr -I <VDR Instance UUID>  --lif <VDR Instance Interface UUID> --proxy-arp -l

• Example outputs when the issue is present are below, these are to show the format of the outputs. For this example we see that 192.168.0.1 is contained within the output of the nest-db command however it is not listed within the net-vdr output, meaning the configuration is present but not realised.
  
  

  /opt/vmware/nsx-nestdb/bin/nestdb-cli --cmd "get vmware.nsx.nestdb.ArpTableConfigMsg" --json --beautify  p
  
                       {
                          "ip" : "192.168.0.1",
                          "mac" : "0",
                          "type" : "PROXY_ARP_ENTRY"        
                       },
                       {
                          "ip" : "192.168.0.2",
                          "mac" : "0",
                          "type" : "PROXY_ARP_ENTRY"
                        }
  
  
  
   net-vdr -I <VDR Instance UUID>  --lif <VDR Instance Interface UUID> --proxy-arp -l
  
  
  DR <UUID> Proxy ARP Information :
  Network           Mac
  192.168.0.2                 <MAC>

• When comparing proxy ARP entries in net-vdr and nest-db, it may appear random as to how many entries are realized once more than 255 are configured. Not all entries beyond 255 are lost, but realization becomes inconsistent.

Environment

VMware NSX 
VMware NSX-T Data Center

Cause

This issue is caused by a limit of ARP proxy, which is capped at 255 ARP proxies per interface. The number of ARP proxies depends on the services using them. For example, a virtual server will create a proxy ARP entry for itself if its IP overlaps with a segment connected to a logical router. If the segments are larger than a /24 or if there are more than 255 overlapping virtual servers, this limitation can be reached.

Resolution

This feature is enhanced in VMware NSX 4.2.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround: If an upgrade to NSX 4.2.2 is not immediately possible, consider one of the following options:

• Architectural: Re-IP the Load Balancer VIP to an address range that does not logically overlap with the backend VM's Layer 2 segment.
• Guest OS Level: Add a persistent /32 static route within the backend VM's guest operating system, forcing the VIP traffic to the default gateway instead of attempting local ARP resolution.