ESXi での仮想マシン ファイル ロックの調査

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:
免責事項：これは英文の記事「Investigating virtual machine file locks on ESXi hosts」の日本語訳です。記事はベストエフォートで翻訳を進めているため、ローカライズ化コンテンツは最新情報ではない可能性があります。最新情報は英語版の記事で参照してください。

• 仮想マシンをパワーオンすると、パワーオン タスクがいつまでも 95% のままとなる。
• 仮想マシンが「無効」状態です。
• 既存の仮想マシン ディスク (VMDK) を、すでにパワーオンした仮想マシンに追加すると、次のエラーで失敗する。
  • Failed to add disk scsi0:1. Failed to power on scsi0:1
  • Cannot power on the virtual machine after deploying it from a template.
    
    
• 仮想マシンのパワーオンが次のエラーで失敗する。
  • Unable to open Swap File
  • Unable to access a file since it is locked
  • Unable to access a file <filename> since it is locked
  • Unable to access Virtual machine configuration
    
    
• /var/log/vmkernel ログ ファイルに、次のようなエントリが表示される。
  WARNING: World: VM xxxx: xxx: Failed to open swap file <path>: Lock was not free
WARNING: World: VM xxxx: xxx: Failed to initialize swap file <path>
   
• 仮想マシンへのコンソールを開くと、次のようなエラーが表示されることがある。
  Error connecting to <path><virtual machine>.vmx because the VMX is not started
 
• 仮想マシンが、vCenter Server と ESXi/ESX ホスト コンソールで、競合する電源状態を報告する。
• （cat or vi などの）テキスト エディタを使用して .vmx ファイルを表示するか開くと、次のようなエラーが報告される。
  cat: can't open '[name of vm].vmx': Invalid argument

Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0.x

Resolution

Purpose of File Locking

To prevent simultaneous changes to critical virtual machines and file systems, the ESXi/ESX host places locks on these files. In certain circumstances, these locks may not be released when the virtual machine is powered off. While the lock remains, the server cannot access the files and the virtual machine cannot be powered on.

The following virtual machine files become locked when they are running:

• VMNAME.vswp
• DISKNAME-flat.vmdk
• DISKNAME-ITERATION-delta.vmdk
• VMNAME.vmx
• VMNAME.vmxf
• vmware.log

Initial Quick Test

• Put DRS into maintenance mode and try to power on the virtual machine on all hosts in the cluster, if it powers on that host it is the host that has the lock.

 

Identifying locked files

1. Power on the virtual machine and note any errors.
2. Connect via ssh to the ESXi host where the virtual machine is located.
3. To find the IP address of the host holding the lock, run vmfsfilelockinfo on the VMDK flat, delta, or sesparse file (for VMFS) or on the .UUID.lck file (for vSAN).
   Enter the following parameters for vmfsfilelockinfo:
   • File to test
   • The user name and password of the vCenter Server (when tracking the MAC address of the ESX host)

Run Example

vmfsfilelockinfo -p /vmfs/volumes/iscsi-lefthand-2/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.10 -u [email protected]

vmfsflelockinfo Version 1.0
Looking for lock owners on "VM1_1-000001-delta.vmdk"
"VM1_1-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['xx:xx:xx:xx:xx:xx']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 0 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Could not get information from Fault domain manager
Connecting to 192.168.1.10 with user [email protected]
Password: xXxXxXxXxXx
----------------------------------------------------------------------
Found 3 ESX hosts from Virtual Center Server.
----------------------------------------------------------------------
Searching on Host 192.168.1.178
Searching on Host 192.168.1.179
Searching on Host 192.168.1.180
MAC Address : xx:xx:xx:xx:xx:xx

Host owning the lock on the vmdk is 192.168.1.180, lockMode : Exclusive

Total time taken : 0.27 seconds.
 
注： 仮想マシンのライフサイクルの中で、いくつかのファイルはロックの状態が遷移します。lock state mode はファイルのロックの種類を示します。

• • mode 0 = no lock
  • mode 1 = is an exclusive lock (vmx file of a powered on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)
  • mode 2 = is a read-only lock (For example on the ..-flat.vmdk of a running virtual machine with snapshots)
  • mode 3 = is a multi-writer lock (For example used for MSCS clusters disks or FT VMs)

4. Run lsof locked_file_name  on the host that holds the lock to find the process that is holding the lock

Run Example

lsof | egrep 'Cartel|VM1_1-000001-delta.vmdk'

Cartel | World name | Type | fd | Description
36202 vmx FILE 80 /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/VM1/VM1_1-000001-delta.vmdk
The Cartel ID holding the lock is 36202.

5. The following command will display a list of active Cartel IDs.

esxcli vm process listThe

output will be as follows:

Alternate_VM27
World ID: 36205
Process ID: 0
VMX Cartel ID: 36202
UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a9 dc 9f bf
Display Name: Alternate_VM27
Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM27/Alternate_VM27.vmx
………

In the above example, the virtual machine with "VMX Cartel ID: 36202" 
holds the lock on VM1_1-000001-delta.vmdk, and the virtual machine is Alternate_VM27. 

6. If no process is listed, use the following script to find the virtual machine:

for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/"  '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei VMDKS_TO_LOOK_FOR ; done

Unlocking

1. Stop the virtual machine that is holding the lock.
2. Reboot the ESXi host that is holding the lock.
   
   Note: If the ESXi host shuts down abruptly due to a hardware failure, you may not be able to reboot the host. To remove the lock on the 'vmx' file in this scenario, see the KB Remove VM vmx file lock .

Remove the .lck file from the VM root directory (NFS only)

1. Stop the virtual machine.
   
   Note : Only delete the .lck file for a powered off virtual machine.

2. Create a directory "bkup" in the VM folder using the command "mkdir bkup" .

3. Move the lock file " lck- #### " using "mv lck-#### bkup" command .

4. 「ls -lah bkup」コマンドを使用してファイルがNavigate to the "bkup" folder and make sure it's in the file list.

Note : VMFS volumes do not have lck files. The locking mechanism for VMFS volumes is handled within the VMFS metadata on the volume.

 

Check the integrity of the virtual machine configuration file (.vmx)

For more information on checking the integrity of virtual machine configuration files, see Verifying ESX/ESXi virtual machine file integrity

. Related information:

• "Cannot open the disk" errors powering on a VM
• "An error occurred while consolidating disks:" deleting one or more snapshots or consolidate disks
• Powering off an unresponsive virtual machine on an ESXi host

Additional Information

• VMware Skyline Health Diagnostics for vSphere - FAQ
• Troubleshooting virtual machine power on task hangs
• Failed to power on virtual machine
• Forcibly Stop a Virtual Machine with ESXCLI - Kill the VM's processes

Note: If a lock is acquired during the switch period between the initial snapshot disk and the base disk, the VM may shut down after consolidation. For more information, seeVirtual Machine shuts down after a disk consolidation due to a locked file