Clearing the ARP cache table in ESXi 7.x

Products

VMware vSphere ESXi

Issue/Introduction

vSphere virtual switches do not hold a "traditional" MAC table (MAC learning provides network connectivity to deployments where multiple MAC addresses are used from one vNIC).

The Source and Destination IP in a RARP packet (in the IP payload) send by the ESXi host are random.

Newer builds of vSphere 7.0 and 8.0 will use IP notation "0.0.0.0" for all virtual switch types.
The IP information has no negative impact on the upstream since the MAC address is the important part.
The RARP packet is purely intended to speed-up MAC learning on the physical switches, which works as intended.

Reference : https://knowledge.broadcom.com/external/article/324542/troubleshooting-virtual-machine-network.html

Environment

VMware vCenter Server 6.7.x
VMware vSphere ESXi 6.7.x
VMware vSphere ESXi 7.0.x
VMware vCenter Server 7.0.x
VMware vSphere ESXi 8.0.x
VMware vCenter Server 8.0.x

Cause

It appears to be a known bug in arp cache service on some physical switches.

Resolution

We can execute this " esxcli network ip neighbor remove " command to clear the ARP cache table.

To clear the ARP cache table in ESXi 7.x, run this command:

esxcli network ip neighbor remove [options]

Where options include:

-i string or --interface-name=string

Where string is the name of the VMkernel network interface from which the neighbor entry must be removed. If this option is not specified, the neighbor is removed from all interfaces.
-a address or --neighbor-addr=address

Where address is the IPv4/IPv6 address of the neighbor. This is mandatory.
-N instance or --netstack=instance

Where instance is the network stack instance. If unspecified, the default netstack instance is used.
-v number or --version=number

Where number is the IP version and can either be 4 or 6. This is mandatory.

For example, to delete the ARP entry for address 10.1##.0.1##:

Connect to the ESXi 7.x host using SSH.
View the current ARP table using this command:

# esxcli network ip neighbor list

You see output similar to:

Neighbor Mac Address Vmknic Expiry State Type ------------ ----------------- ------ ------- ----- ----- 10.131.0.103 xx:xx:xx:xx:xx:xx vmk0 908 sec Unknown 10.131.0.179 xx:xx:xx:xx:xx:xy vmk0 1062 sec Unknown
To delete the ARP entry for address 10.131.0.103, run one of these commands:
- # esxcli network ip neighbor remove -v 4 -a 10.131.0.103
- # esxcli network ip neighbor remove --version=4 --neighbor-addr=10.131.0.103
View the ARP table again using this command:

# esxcli network ip neighbor list

You see output similar to:

Neighbor Mac Address Vmknic Expiry State Type ------------ ----------------- ------ ------- ----- ----- 10.131.0.179 xx:xx:xx:xx:xx:xz vmk0 750 sec Unknown

Additional Information

Please check the Junos version and Juniper switch model being used. We might be hitting this known issue with flooding in a VLAN although the MAC address is learnt - https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1321612

Could you try to upgrade to a fixed version and confirm if things look better? - Junos 15.1X53-D58, 18.1R1 or the latest JTAC recommended release should do - https://kb.juniper.net/InfoCenter/index?page=content&id=KB21476

Cisco Bug : https://bst.cisco.com/quickview/bug/CSCwd97579

Products (8)

Cisco Catalyst 3650 Series Switches, Cisco Catalyst 3850 Series Switches, Cisco Catalyst 9200 Series Switches, Cisco Catalyst 9300 Series Switches, Cisco Catalyst 9400 Series Switches, Cisco Catalyst 9500 Series Switches, Cisco Catalyst 9500H Series Switches, Cisco Catalyst 9600 Series Switches

Known Affected Release

17.6.4

Description (partial)

Symptom: A Catalyst 9300 switch may not punt a Gratuitous ARP packet to the CPU. This results in ARP entries not updating if the MAC address changes for the IP, resulting in black-holing of traffic The G-ARP packet can be observed via embedded packet capture utility on the interface where it is received but is not observed in a FED punt debug indicating the packet is not sent to the CPU Conditions: G-ARP packet coming in L2 interface when MAC address is different from current ARP entry MAC address This can be observed when multiple devices share a virtual IP in an active/standby setup and rely on G-ARP to update the MAC on the ARP entry when failovers are performed This behavior has been observed on 17.6.4