RARP (Reverse ARP) broadcast packets are sent to the switch which the edge uplinks connected to when NSX Edge Nodes are rebooted/powered on, vMotioned or Storage vMotioned
search cancel

RARP (Reverse ARP) broadcast packets are sent to the switch which the edge uplinks connected to when NSX Edge Nodes are rebooted/powered on, vMotioned or Storage vMotioned

book

Article ID: 322563

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

  • This article is to provide information about RARP packet behaviour during VM reboot/power on or vMotion of guest VMs belonging to a trunk VLAN
  • The Edge node's uplink is configured with a trunk VLAN (0-4094) for connectivity to the physical switch.
  • 10,000+ RARP packets are broadcast to the physical switch via the edge node uplink interface in a period of 400 milliseconds, when performing edge reboot/power on rr vMotion/Storage vMotion of the edge node. 
  • This leads the physical switch to detect a broadcast storm if it has a storm control configuration like "storm-control broadcast level pps 10k".

Environment

VMware NSX-T Data Center
VMware NSX 4.x

Cause

  • RARP packets are sent to the Edge node's uplink physical switch, when during vMotion of an NSX Edge Node, so that physical switch can learn MAC address of NSX Edge Node's fp-ethX (VTEP) interface.
  • This behaviour is also during Edge Node reboot or power on or any other operations where its vNICs are activated.
  • Without these RARP packets, packets destined to the NSX Edge Node VTEP interface could continue to be forwarded to the ESXi host where the edge node resided before the vMotion.
  • This would cause momentary disruption to North/South traffic until the NSX Edge Node VTEP MAC is learned correctly on the upstream switch/switches.This could typically last 10 minute until the old ARP entry for Edge expires and restores when a new ARP request is initiated and a new reply from the Edge node is received.
  • With regards to Storage vMotion, the port of NSX Edge Node is disconnected and enabled with vMotion flag. Hence RARP packets are sent out, too.
  • 4 RARP packets are sent per VLAN. Hence in the worst case, with Trunk VLAN range (0-4094), 4 * 4095 = 16,380 RARP packets are expected to go out of the uplink in a period of 400ms.

Resolution

This is expected behaviour in VMware NSX.

Workaround:
Reduce the number of Trunk VLANs to ones which are only necessary (e.g. 0-4094 to 0-1024) based on your usage. RARP Request packets during Edge Node reboot, power on, vMotion or Storage vMotion will be reduced accordingly.