ESXi host initiates ARP Broadcast storm to NFS server
Article ID: 337889
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
- Physical network is saturated by large number of ARP Requests.
- Capturing a trace on the ESXi host using tcpdump command confirms that these ARP Requests are created from a number of ESXi hosts.
- Running the tcpdump command displays an output similar to:
13:09:50.319752 ARP, Request who-has w.x.y.z tell xx.xx.xx.xx, length 46
13:09:50.319764 IP truncated-ip - 94 bytes missing! xx.xx.xx.xx.2609655792 > w.x.y.z.2049: 120 getattr [|nfs]
13:09:50.319842 ARP, Request who-has w.x.y.z tell xx.xx.xx.xx, length 46
13:09:50.319960 IP w.x.y.z.2049 > 10.22.234.37.771: Flags [F.], seq 1, ack 125, win 32705, options [nop,nop,TS val 3819598309 ecr 2678248935], length 0
13:09:50.319979 IP xx.xx.xx.xx.771 > w.x.y.z.2049: Flags [.], ack 2, win 512, options [nop,nop,TS val 2678248935 ecr 3819598309], length 0
13:09:50.319995 ARP, Request who-has w.x.y.z tell 10.22.232.17, length 46
13:09:50.320008 IP xx.xx.xx.xx.771 > w.x.y.z.2049: Flags [R.], seq 125, ack 2, win 512, options [nop,nop,TS val 2678248935 ecr 3819598309], length 0
13:09:50.320024 ARP, Request who-has w.x.y.z tell xx.xx.xx.xx, length 28
13:09:50.320121 ARP, Reply w.x.y.z is-at xx:xx:xx:xx:xx:xx, length 46
13:09:50.320139 ARP, Reply w.x.y.z is-at xx:xx:xx:xx:xx:xy, length 46
- In the /var/log/vmkernel.log file (ESXi log file), you see error similar to:
WARNING: NFS: 322: Lost connection to the server w.x.y.z mount point /vol/....
Environment
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.1
Cause
This issue occurs if incorrect steps are used when removing a NFS datastore from the ESXi host.
Resolution
The ESXi host is able to resolve the IP to the correct MAC, but the ARP Request Network storm continues.
To resolve this issue, unmount the datastore by running these commands on each host that has access to the NFS server with IP address w.x.y.z:
To resolve this issue, unmount the datastore by running these commands on each host that has access to the NFS server with IP address w.x.y.z:
- Run this command to list mounted datastore:
esxcfg-nas -l
You see output that displays the volume that was incorrectly removed as mounted, but unavailable.
- Unmount the datastore on each host. For more information, see Unmounting a LUN or detaching a datastore/storage device from multiple VMware ESXi 5.x hosts (2004605). For more information on unmounting NFS datastores on an ESXi host, see Remounting a disconnected NFS datastore from the ESXi/ESX command line (1005057).
If the problem still persists after performing the preceding step, reboot the affected host to clear all internal references to the NFS datastore. All virtual machines in the host must be shut down or migrated off the host.
Additional Information
Feedback
Yes
No
Powered by
