Cannot power on a virtual machine located in an NFS datastore

Products

• VMware vSphere ESXi

Issue/Introduction

You may see the below:

• You are unable to power on a virtual machine located in an NFS datastore.
• Powering on a virtual machine that is located in a NFS datastore fails.
• You see an error similar to:
  
  Failed to lock the file. Cannot open the disk '/vmfs/volumes/UUID/VM_Name/VM_Name.vmdk' or one of the snapshot disks it depends on.

• The /vmfs/volumes/datastore_name/virtual_machine_folder/vmware.log file, contains entries similar to:
  
  vmx| DISKLIB-VMFS : "/vmfs/volumes/UUID/VM_Name/VM_Name-flat.vmdk" : failed to open (Failed to lock the file): AIOMgr_Open failed. Type 3
  vmx| DISK: Cannot open disk "/vmfs/volumes/UUID/VM_Name/VM_Name.vmdk": Failed to lock the file (16392).
  vmx| Msg_Post: Error
  vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/UUID/VM_Name/VM_Name.vmdk' or one of the snapshot disks it depends on.
   
• The /var/log/vmkernel file contains entries similar to:
  
  NFSLock: 2819: failed to get lock on file VM_Name.vmdk 0x000000000000 on ip> (ip): Busy
  WARNING: NFSLock: 2128: disk is being locked by other consumer
  NFSLock: 2819: failed to get lock on file <VM_Name>-flat.vmdk 0x000000000000 on ip (ip): Busy

Environment

VMware vSphere ESXi 5.1
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

This issue occurs when the .lck-xxx file located in the virtual machine folder holds a lock on the vmdk file. When an ESXi/ESX host accesses a virtual machine disk file on a NFS based datastore a .lck-xxx lock file is generated in the same directory as the disk file. This file prevents other ESXi/ESX hosts from accessing the virtual disk file.

Resolution

Check the file NFS file locks using the Understanding the NFS .lck lock file to understand the ESX host and NFS filename it refers to KB to identify the file locks.

 

To resolve this issue, delete the .lck-xxxx file located in the virtual machine folder.

Caution: Ensure the vmdk files are not in use by another virtual machine or backup process before performing these steps. Removing a valid .lck can cause instability.

To delete the .lck-xxxx file:

1. Ensure the virtual machine is powered off.
2. Connect to the ESXi/ESX host using Secure Shell (SSH). 
3. Go to the virtual machine folder, run the command:
   
   cd /vmfs/volumes/datastore_name/virtual_machine_folder
    
4. List all files including the hidden files, run the command:
   
   ls -la
    
5. Remove the lock file, run the command:
   
   rm .lck-xxxx
   
   Where xxxx is a sequence of numbers. For example, lck-001455721.
   
   Note: Ensure the virtual machine is not running.
    
6. Power on the virtual machine.

Note: When the virtual machine is powered on, you see a new .lck-xxxx file generated in the virtual machine folder. This is expected behavior.
 

If this issue persists or no lock files are present to apply this resolution to, file a Support Request with VMware Support and note this KB article ID (2037507) in the problem description. For more information, see Creating and managing Broadcom support cases.

 

Additional Information