NSX-T Edge receives a high volume of ARP requests from ESXi Transport Nodes

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• NSX-T Data Center 3.1.0 and 3.1.1.
• The Edge node is receiving a high rate of a ARP Request from the ESXi transport nodes, requesting to resolve 169.254.0.x.
• On the Edge node, /var/log/syslog has a large number of entries tracking these ARP replies, e.g.

  2021-06-04T18:46:02.359Z Manager1 NSX 6622 SWITCHING [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="arp" level="INFO"] ARP reply sent to 00:50:56:##:##:4a for 169.254.0.2 from 02:50:##:##:53:00 on lrouter port 0c456328-70e5-####-####-########3b1
  
  2021-06-04T18:46:02.361Z Manager1 NSX 6622 SWITCHING [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="arp" level="INFO"] ARP reply sent to 00:50:56:##:##:4a for 169.254.0.2 from 02:50:##:##:53:00 on lrouter port 0c456328-70e5-####-####-########3b1
  
  2021-06-04T18:46:02.361Z Manager1 NSX 6622 SWITCHING [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="arp" level="INFO"]  message repeated 2 times: [ARP reply sent to 00:50:##:##:76:4a for 169.254.0.2 from 02:50:##:##:53:00 on lrouter port 0c456328-70e5-####-####-########3b1]

• If the ARP rate is sufficiently high, the alarm "Edge NIC Out of Receive Buffer" may be triggered on the Edge node.
• If the ARP rate is sufficiently high, high datapath CPU may be observed against some cores even though the packet rate is not high
  

  e.g.
  > get dataplane cpu stats
  
  Core      : 2
  Crypto    : 0 pps
  Intercore : 0 pps
  Kni       : 0 pps
  Rx        : 18460 pps <<
  Slowpath  : 0 pps
  Tx        : 18490 pps
  Usage     : 86%         <<

Environment

VMware NSX-T Data Center

Cause

Incorrect logic in the DR of the ESXi hosts resulted in excessive ARP requests, for the SR/DR IP, being sent on the transit link to the Edge nodes.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.1.2 available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

If you believe you have encountered this issue and are unable to upgrade, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.