RARP (Reverse ARP) broadcast packets are sent to the switch which the edge uplinks connected to when NSX-T Edge Nodes are rebooted/powered ON, vMotion'ed or Storage vMotion'ed

Products

• VMware NSX Networking

Issue/Introduction

This article is to provide information about RARP packet behavior during VM reboot/power ON or vMotion of guest VMs belonging to a trunk VLAN

Symptoms:

• The edge node uplink is configured with a trunk VLAN (0-4096) for connectivity to the physical switch.
• 10,000+ RARP packets are broadcast to the physical switch via the edge node uplink interface in a period of 400 milliseconds, when performing edge reboot/power ON Or vMotion/Storage vMotion of the edge node. 
• This leads the physical switch to detect a broadcast storm if it has a storm control configuration like "storm-control broadcast level pps 10k".

Environment

VMware NSX-T Data Center

VMware NSX-T Data Center 3.x

VMware NSX 4.x

Cause

• RARP packets are sent to the edge nodes uplink physical switch, when during vMotion of an NSX-T Edge Node, so that physical switch can learn MAC address of NSX-T Edge Node's fp-ethX (VTEP) interface.
• This behaviour is also during Edge Node reboot or power ON or any other operations where its vNICs are activated.
• Without these RARP packets, packets destined to the NSX-T Edge Node VTEP interface could continue to be forwarded to the ESXi host where the edge node resided before the vMotion.
• This would cause momentary disruption to North/South traffic until the NSX-T Edge Node VTEP MAC is learned correctly on the upstream switch/switches.
• With regards to Storage vMotion, the port of NSX-T Edge Node is disconnected and enabled with vMotion flag. Hence RARP packets are sent out, too.
• 4 RARP packets are sent per VLAN. Hence in the worst case, with Trunk VLAN range (0-4094), 4 * 4095 = 16,380 RARP packets are expected to go out of the uplink in a period of 400ms.

Resolution

This is expected behavior in NSX-T Data Center.

Workaround:
Reduce the number of Trunk VLANs to ones which are only necessary (e.g. 0-4094 to 0-1024) based on your usage. RARP Request packets during Edge Node reboot, power ON, vMotion or Storage vMotion will be reduced accordingly.