Powering on a virtual machine on NFS or trying to remove an NFS Datastore fails with errors "Unable to access a file since it is locked" or "Resource is in use"

Products

• VMware vSphere ESXi

Issue/Introduction

This article is specific to NFS datastores. For similar error on VMFS datastores, see Investigating virtual machine file locks in ESX/ESXi (10051).

Symptoms:

You are experiencing one or more of these issues:

• Powering on a virtual machine fails
• If you try to power on a virtual machine, you see the error:
  
  Unable to access a file since it is locked
  
• Removing an NFS datastore fails
• If you try to remove an NFS datastore, you see the error:
  
  Resource is in use

Environment

VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.0
VMware ESX Server 3.0.x
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Embedded
VMware ESXi 4.0.x Installable
VMware ESX 4.1.x
VMware ESXi 3.5.x Embedded
VMware ESX 4.0.x
VMware ESXi 3.5.x Installable
VMware vSphere ESXi 5.5
VMware ESX Server 3.5.x

Resolution

NFS locking on ESX uses its own locking protocol, not the Network Lock Manager (NLM) protocol. NFS locks are implemented by creating lock files on the NFS server.

Lock files are named .lck-fileid, where fileid is the value of the fileid field returned from a GETATTR request.

To power on a virtual machine and remove a locked NFS datastore:

1. Log in to the ESX host using an SSH client. For more information, see Connecting to an ESX host using a SSH client (1019852).
2. Navigate to the directory where the virtual machine resides.
3. Determine the name of the NFS lock (.lck) file with the command:
   
   ls -la
   
   The output appears similar to:
   
   total 47383432
   drwxr-xr-x 1 root root 4096 Jul 17 07:28 .
   drwxr-xr-x 1 root root 4096 Jul 14 08:37 ..
   -rwxrwxrwx 1 root root 84 Jul 17 2009 .lck-1ab0140000000000
   
   
4. Determine the ESX host that created the .lck file with the command:
   
   strings .lck_filename
   
   The output appears similar to:
   
   esxhost2.domain.local
   
   Note: In an ESXi host, use the cat or od command against the lock file. The output contains additional characters with the hostname. For example: _³#Í ÿÿÿÿESX_HOSTNAME_123JÙ/yVr, where, ESX_HOSTNAME_123 is the hostname.
   
   
5. Connect to the ESX host that created the .lck file (esxhost2.domain.local, in this example) using SSH.
6. Obtain a list of virtual machine worlds which are running on the host using one of the methods in Mapping a virtual machine world number to a virtual machine name (1001101).
   
7. If the virtual machine is running, connect to the ESX host using vSphere Client and verify that it is powered on.
8. If you want to remove a NFS datastore, power off the virtual machine and remove the datastore.
   
   Note: If a virtual machine does not show as running in the vSphere Client or you are unable to stop it, terminate the virtual machine. For more information, see:
   • Powering off a virtual machine (1004340) (for ESX)
   • Powering off a virtual machine on an ESXi host (1014165) (for ESXi)

If you do not know which virtual machines are running on the NFS, you can find all the locked files inside the NFS datastore with the command:

# find /vmfs/volumes/<NFSDatastoreName>/ -iname ".lck*"

For example:

# find /vmfs/volumes/dc03-nfs/ -iname ".lck*"

The output appears similar to:

/vmfs/volumes/dc03-nfs/DebianVM/.lck-6604000000000000
/vmfs/volumes/dc03-nfs/DebianVM/.lck-7004000000000000
/vmfs/volumes/dc03-nfs/DebianVM/.lck-7204000000000000
/vmfs/volumes/dc03-nfs/New Virtual Machine/.lck-d71d000000000000
/vmfs/volumes/dc03-nfs/New Virtual Machine/.lck-da1d000000000000

Note: If you do not have any virtual machines on the NFS mount, the command # find /vmfs/volumes/<NFSDatastoreName>/ -iname ".lck*" may not help you.

If your virtual machines have ISO images mounted from the NFS, they have to be removed from the virtual machine settings to release the locks.

For example, run the command:

# find /vmfs/volumes/ISOs/ -iname ".lck*"

The output appears similar to:

/vmfs/volumes/ISOs/OS/redhat/.lck-1340d10000000000
/vmfs/volumes/ISOs/OS/redhat/.lck-1440d10000000000
/vmfs/volumes/ISOs/OS/redhat/.lck-1540d10000000000
/vmfs/volumes/ISOs/OS/redhat/.lck-0540d10000000000
/vmfs/volumes/ISOs/OS/redhat/.lck-0640d10000000000
/vmfs/volumes/ISOs/OS/redhat/.lck-0740d10000000000

Note: The output indicates that the ISOs are locked.

To find the virtual machines that have ISOs mounted, run the command:

# grep -i ".iso" /vmfs/volumes/*/*/*vmx

The output appears similar to:

/vmfs/volumes/4b73f167-c9f86304-50e6-001517ab928b/FreeNAS-RDM's/FreeNAS-RDM's.vmx:ide1:0.fileName = "/vmfs/volumes/0bd57674-c63b0c55/FreeNAS-i386-0.69RC1.3991.iso"

Additional Information

Mapping a virtual machine world number to a virtual machine name
Investigating virtual machine file locks on ESXi