ESXi host initiates ARP Broadcast storm to NFS server

Products

• VMware vSphere ESXi

Issue/Introduction

• Physical network is saturated by large number of ARP Requests.
• Capturing a trace on the ESXi host using tcpdump command confirms that these ARP Requests are created from a number of ESXi hosts.
• Running the tcpdump command displays an output similar to:

13:09:50.319752 ARP, Request who-has #.#.#.# tell #.#.#.#, length 46
13:09:50.319764 IP truncated-ip - 94 bytes missing! #.#.#.#.2609655792 > #.#.#.#.2049: 120 getattr [|nfs]
13:09:50.319842 ARP, Request who-has #.#.#.# tell #.#.#.#, length 46
13:09:50.319960 IP #.#.#.#.2049 > #.#.#.#.771: Flags [F.], seq 1, ack 125, win 32705, options [nop,nop,TS val 3819598309 ecr 2678248935], length 0
13:09:50.319979 IP #.#.#.#.771 > #.#.#.#.2049: Flags [.], ack 2, win 512, options [nop,nop,TS val 2678248935 ecr 3819598309], length 0
13:09:50.319995 ARP, Request who-has #.#.#.# tell #.#.#.#, length 46
13:09:50.320008 IP #.#.#.#.771 > #.#.#.#.2049: Flags [R.], seq 125, ack 2, win 512, options [nop,nop,TS val 2678248935 ecr 3819598309], length 0
13:09:50.320024 ARP, Request who-has #.#.#.# tell #.#.#.#, length 28
13:09:50.320121 ARP, Reply #.#.#.# is-at #:#:#:#:#:#:#, length 46
13:09:50.320139 ARP, Reply #.#.#.# is-at #:#:#:#:#:#:#, length 46

• In the ESXi - /var/log/vmkernel.log file, the following log message appears:

WARNING: NFS: 322: Lost connection to the server #.#.#.# mount point /vol/....

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This issue occurs if incorrect steps are used when removing a NFS datastore from the ESXi host.

Resolution

The ESXi host is able to resolve the IP to the correct MAC, but the ARP Request Network storm continues.

To resolve this issue, unmount the datastore by running these commands on each host that has access to the NFS server with IP address #.#.#.#:

1. List mounted datastores:
   
   esxcfg-nas -l
   
   An output will display the volume that was incorrectly removed as mounted, but unavailable.
   
   
2. Unmount the datastore on each host. For more information, see How to detach a LUN device from ESXi hosts
3. For more information on unmounting NFS datastores on an ESXi host, see Remounting a disconnected NFS datastore from the ESXi command line
4. If the problem still persists after performing the preceding step, reboot the affected host to clear all internal references to the NFS datastore. All virtual machines in the host must be shut down or migrated off the host.