This issue is resolved in ESX 4.0 Update 2. You can download ESX 4.0 Update 2 from the vSphere Download Center.
Prior to ESX 4.0 Update 2, this issue occurs when different ESX hosts have NICs with the same MAC address. The MAC address of one NIC is used to construct the System ID of the host. Hosts use this ID to identify what is performing operations on the metadata of the datastore and therefore should be unique.
# esxcfg-info -y | grep -i "System UUID"
|----System UUID.................................................4b27ad9b-2120-4013-99ac- 00237da6fc54
esxcfg-nics -l
# esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 42:00.00 bnx2 Up 1000Mbps Full xx:xx:xx:xx:xx:xx 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic1 44:00.00 bnx2 Up 1000Mbps Full xx:xx:xx:xx:xx:xx 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic2 4b:00.00 e1000e Up 1000Mbps Full xx:xx:xx:xx:xx:xx 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic3 4b:00.01 e1000e Down 0Mbps Half xx:xx:xx:xx:xx:xx 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
For ESX 3.x, use this command to find the MAC address:
# ifconfig | grep -i HWaddr
inet addr:xx.xx.xx.xx Mask:255.0.0.0
vmnic0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
vmnic1 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
vmnic2 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
vmnic3 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
vswif0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx