Tier-1 Gateway Cannot Reach VM Because ARP Entry Resolves to Wrong MAC

Products

• VMware NSX

Issue/Introduction

A virtual machine on a routed segment cannot communicate with its Tier-1 gateway, while other VMs on the same subnet work normally.

Typical symptoms:

• ICMP/HTTP from the affected VM to the Tier-1 gateway fails.
• The VM’s ARP table shows a valid gateway IP/MAC.
• On the Tier-1 gateway, the neighbor entry for the VM’s IP shows a MAC address that does not match the VM’s vNIC MAC.
• You can use this command to check the FIB/ARP records on the Tier-1 gateway:

  # edge-appctl -t /var/run/vmware/edge/dpd.ctl lrouter/show all | python -m "json.tool"

      {
          "uuid": "########-####-####-####-############",
          "name": "TIER1-SR-NAME",
          "type": "SERVICE_ROUTER_TIER1",
          ...
          "fib": [
              ...
              {
                  "prefix": "##.##.##.##/32",
                  "type": "neighbor",
                  "next-hops": [
                      {
                          "gw-ip": "##.##.##.##",
                          "ifuuid": "########-####-####-####-############",
                          "ifuid": ###,
                          "l2-state": "reachable",
                          "gw-mac": "##:##:##:##:##:##", <----- MAC address from other vendor. In a correct environment, this should be a VMware MAC address (starting with 00:50:56).
                          "nh-ref-count": 1,
                          "nh-hit-flag": true,
                          "nh-path-down": false
                      }
                  ]
              },
          ...
          "arp": [
             ...
             {
                  "ifuuid": "########-####-####-####-############",
                  "ip": "##.##.##.##",
                  "vlan": ####,
                  "mac": "##:##:##:##:##:##", <----- MAC address from other vendor. In a correct environment, this should be a VMware MAC address (starting with 00:50:56).
                  "state": "reach",
                  "mheld_cnt": 0,
                  "timeout": 326,
                  "last event": "YYYY-MM-DDTHH:MM:SS.sss",
                  "stats": {
                      "pkt_out": 4,
                      "icmp_out": 0,
                      "pkt_out_fail": 0,
                      "solicit_out": 35366,
                      "solicit_out_fail": 0,
                      "solicit_in": 35363,
                      "unsolicit_in": 3,
                      "ip_solicit_out": 0,
                      "ip_solicit_out_fail": 0,
                      "announce_out": 0,
                      "announce_out_fail": 0
                  }
              },

Environment

VMware NSX-T Data Center
VMware NSX

Cause

The Tier-1 gateway has learned an incorrect MAC address for the VM’s IP due to ARP interference on the Layer-2 segment, for example:

• Another host using the same IP (IP conflict).
• A firewall/router/load balancer configured with Proxy ARP, VIP, or NAT that answers ARP for the VM’s IP.

Resolution

1. Confirm the wrong neighbor entry

• On the Tier-1 gateway, check the neighbor/ARP table and verify the VM’s IP is mapped to an incorrect MAC (not the VM’s vNIC MAC).

2. Identify and fix the device answering ARP

• Capture ARP traffic for the VM’s IP on the segment and see which device replies.
• On that device, remove or correct any duplicate IP, Proxy ARP, or VIP/NAT configuration that advertises the VM’s IP.