VMFS datastore access issues with 0 Bytes size and lock errors

Products

• VMware vSphere ESXi

Issue/Introduction

• Some VMFS datastores are inaccessible on some ESXi hosts.
  
  
  
• On these ESXi hosts, the datastores report 0 Bytes in size via the vSphere Client.
  
  
  
• /var/log/vmkernel.log reports repeated entries for the LUN backing the datastores similar to: 

VM: 7379: Device <naa.################################:1> locked by <ESXi_UUID> at <epoch timestamp> (29 tries left)
 
All hosts reports report that the devices are locked by the same ESXi host (same ESXi  UUID) and that the locks were placed at the same time (same epoch timestamp)  

• /var/log/vobd.log reports:

[vmfsCorrelator] 1121164524475us: [vob.vmfs.error.volume.is.locked] The volume on the device naa.################################:1 locked, possibly because some remote host encountered an error during a volume operation and could not recover.

Environment

VMware vSphere ESXi

Cause

For the host reported to be holding the locks, a storage issue at the time indicated by the timestamp prevented locks the host held at that time from being released. 

Resolution

Reboot the host holding the locks to clear the locks.

Note:
An ESXi host UUID has the format: ########-########-####-############. The last twelve digits correspond to the digits of the MAC address of a NIC on the ESXi host without the usual MAC format (##:##:##:##:##:##).

Additional Information

Epoch time is the time measured in seconds from 1 January 1970 00:00:00 UTC.