Cannot power on a virtual machine located in an NFS datastore

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• You are unable to power on a virtual machine located in an NFS datastore.
• Powering on a virtual machine that is located in a NFS datastore fails.
• You see an error similar to:
  
  Failed to lock the file. Cannot open the disk '/vmfs/volumes/UUID/VM_Name/VM_Name.vmdk' or one of the snapshot disks it depends on.

• The /vmfs/volumes/datastore_name/virtual_machine_folder/vmware.log file, contains entries similar to:
  
  vmx| DISKLIB-VMFS : "/vmfs/volumes/UUID/VM_Name/VM_Name-flat.vmdk" : failed to open (Failed to lock the file): AIOMgr_Open failed. Type 3
  vmx| DISK: Cannot open disk "/vmfs/volumes/UUID/VM_Name/VM_Name.vmdk": Failed to lock the file (16392).
  vmx| Msg_Post: Error
  vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/UUID/VM_Name/VM_Name.vmdk' or one of the snapshot disks it depends on.
   
• The /var/log/vmkernel file contains entries similar to:
  
  NFSLock: 2819: failed to get lock on file VM_Name.vmdk 0x000000000000 on ip> (ip): Busy
  WARNING: NFSLock: 2128: disk is being locked by other consumer
  NFSLock: 2819: failed to get lock on file <VM_Name>-flat.vmdk 0x000000000000 on ip (ip): Busy

Environment

VMware vSphere ESXi 5.1

Cause

This issue occurs when the .lck-xxx file located in the virtual machine folder holds a lock on the vmdk file. When an ESXi/ESX host accesses a virtual machine disk file on a NFS based datastore a .lck-xxx lock file is generated in the same directory as the disk file. This file prevents other ESXi/ESX hosts from accessing the virtual disk file.

Resolution

To resolve this issue, delete the .lck-xxxx file located in the virtual machine folder.
 

Caution: Ensure the vmdk files are not in use by another virtual machine before performing these steps. Removing a valid .lck can cause instability.

To delete the .lck-xxxx file:

1. Ensure the virtual machine is powered off.
2. Connect to the ESXi/ESX host using Secure Shell (SSH). 
3. Go to the virtual machine folder, run the command:
   
   cd /vmfs/volumes/datastore_name/virtual_machine_folder
    
4. List all files including the hidden files, run the command:
   
   ls -la
    
5. Remove the lock file, run the command:
   
   rm .lck-xxxx
   
   Where xxxx is a sequence of numbers. For example, lck-001455721.
   
   Note: Ensure the virtual machine is not running.
    
6. Power on the virtual machine.

Note: When the virtual machine is powered on, you see a new .lck-xxxx file generated in the virtual machine folder. This is expected behavior.
 

If this issue persists or no lock files are present to apply this resolution to, file a Support Request with VMware Support and note this KB article ID (2037507) in the problem description. For more information, see Creating and managing Broadcom support cases.

 

Additional Information