ロックされた仮想ディスクが原因である問題のトラブルシューティング
search cancel

ロックされた仮想ディスクが原因である問題のトラブルシューティング

book

Article ID: 327410

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article summarizes how to troubleshoot issues based on locked virtual disks. This is a recurring problem, whether you use a snapshot-based backup solution, an unsupported disk format, or due to general storage issues.

Because there are multiple possible causes of locking, this article walks you through a step-by-step process at several points where you can successfully complete the troubleshooting. These points will be marked as "You should be able to integrate/power on". If reaching these points does not resolve the issue, proceed to the next step.

Disclaimer: This is a Japanese translation of the article "Troubleshooting issues resulting from locked virtual disks (2107795)". Because articles are translated on a best-effort basis, localized content may not be up-to-date. Please refer to the English version of the article for the latest information.

 

  • The virtual machine fails to power on
  • You see an error similar to the following:
     
    • Unable to access a file filename since it is locked
    • Unable to access virtual machine configuration
  • Virtual machine snapshot consolidation fails
An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED
  • 次のようなエラーが表示される。

    An error occurred while consolidating disks: msg.fileio.lock.
  • After a successful snapshot-based backup, the Virtual Machine Summary tab displays a message similar to the following:

    Virtual Disk Consolidation is required.


Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 6.0

Cause

この問題は、次のようなさまざまな理由で発生します。
  • 一般的な理由は、パワーオン状態の仮想マシンに、読み取りおよび書き込みアクセスを行うためにそれらを所有する ESXi ホストによって使用されているすべてのファイルに対するロックが含まれることです。
  • その他のロックは、バックアップ プロセス中にスナップショット ベースのバックアップ アプライアンスにディスクをホットアドすることにより発生する場合があります。
  • ロックの作成や仮想マシンの起動の失敗は、サポートされていないディスク フォーマットを使用した場合、またはロックがすでに存在していた場合に発生する可能性があります。

Resolution

Troubleshooting Locked Virtual Disks on VMFS Volumes

1 -
Snapshot and power-on operations that write

details to the log are located in /vmfs/volumes/<vm-datastore>/<vm-name>/vmware.log File (snapshot only if the virtual machine is already powered on).
If there is an integration issue and the virtual machine is powered on, trigger a consolidation task to write a new set of log entries to help troubleshoot the issue.
If the virtual machine fails to power on, skip this item and III(a).

2 - To identify the lock on the disk, you need to clear all possible locks on the disk.

A powered-on virtual machine is protected by an exclusive lock through the owning ESXi host. If applicable, power off the virtual machine.
All remaining locks are considered unexpected and should be investigated.

A - Find locked files
  1. Using SSH, log in as root to the ESXi host that is running the virtual machine. Navigate to /vmfs/volumes/vm-datastore/vm-name/.
  2. Open the /var/log/vmware.log file to see the last integration/power-on process.
  3. There should be an error or warning about at least one file that cannot be accessed due to a locking problem.
  4. Depending on the type of problem, the log messages vary.
If you're having trouble finding the message in question, skip this part and go to III(b).

B - Find locked files (b)
  1. Navigate to the virtual machine's directory and run the command vmkfstools on the corresponding descriptor file.
  2. Run the command vmkfstools -qv10 against the currently mounted snapshot (check the disks in the vCenter Server virtual machine settings to see which ones are mounted).
  3. If you have more than one virtual disk, run the following command for a snapshot of each currently mounted disk:


Example:
vmkfstools -qv10 vm-disk-000009.vmdk

This displays the actual snapshot chain from the selected snapshot to the flat disk.

Note: If this output returns a snapshot chain error, the consolidation issue is caused by a broken chain of snapshots. For information on how to fix it, see the corresponding Knowledge Base article.

Make a note of these files. It is necessary to check further. The files found in the log (step III(a)) should be included in this list.
If you don't have snapshots, you only need to check the flat disk locks. So, you can skip the above part. Then, use the command vmkfstools -D on all the files found in the log for which you want to check the lock.


If you skipped step III, run the following command for all files returned by vmkfstools -qv10:



Example:
vmkfstools -D vm-disk-flat.vmdk
vmkfstools -D vm-disk-000001-delta.vmdk

This returns the MAC address (as part of the UUID) that identifies the locking ESXi host. For more information, see Virtual Machine Won't Power On Because Files Are Locked (1033280).

Example:
gen 2021, mode 1, owner 543d375e-1db58166-0bd6-002219a74751 mtime 691807

mode 1 is an exclusive read/


In this case, run the following command for each ESXi host to check if the virtual disk is connected to another powered-on virtual machine or if the virtual machine is powered off.

grep -i <diskname> /vmfs/volumes/*/*/*.vmx

mode 2 or RO Owner indicates a read-only lock, possibly because the disk is still attached to the backup appliance. Unless this is the most recent snapshot that requires read/write access, this does not prevent power-on, because an existing read-only lock can coexist with another read-only lock. However, the integration is hampered by the need for write access during the integration process.

Run the following command to display the MAC address of the current ESXi host:

esxcfg-nics -lIf

the MAC address in question is not from this ESXi host, check for a different host (vCenter Server > Configuration> Network Adapter)


Note: The MAC address only identifies the host that is causing the lock and is not mistaken for a reference to the actual NIC. Therefore, even if this NIC is not connected to the network, you can still find vmnic0 that is locked.

If you are using a backup solution and this appliance is running on the ESXi host that you found in the above process, power it off, remove the virtual disk from its configuration, or both.


You should be able to integrate/power on.

C - Find the locking process/service caused


by a hot add or another running virtual machine accessing the same disk, if the above troubleshooting does not resolve the issue, There may be a different type of lock.


Do the following:
  1. Connect to the ESXi host identified by the MAC address returned in the last step.
  2. Run the following command to find the process that is dealing with the lock:

    ps | grep -i vm-name
     
  3. If this command returns non-empty output, there is a process that is locking the disk. If not, skip to step b.
  4. In the output above, find vmm0 and write down the ID (at the beginning of the line). Run the following command:

    esxcli vm process list | grep -i <ID> -B1

    This returns the virtual machine corresponding to the process ID. This virtual machine is under lock.
     
  5. Remove the disk from this virtual machine or power off the virtual machine. You should be able to integrate/power on.
  6. Run the following command to find the task/service that is dealing with locking:

    lsof | grep -i <vm-name>

    If this returns output, there is a service or task that is locking the disk. This occurs only on the same ESXi host to which the virtual machine is registered.
     
  7. You can run the following command to get more information about the task:

    vim-cmd vmsvc/getallvms | grep -i vm-name
     
  8. Make a note of the ID you get in the output (the number at the beginning of the line) and run the following command:

    The output of vim-cmd vmsvc/get.tasklist <ID>

    looks like this:

    vim-cmd vmsvc/get.tasklist 109
    (ManagedObjectReference) [
    'vim.Task:haTask-109-vim.vm.Snapshot.remove-92744898'
    ]

    Make a note of the task ID after the ":".
     
  9. 次のコマンドを実行します。

    vim-cmd vimsvc/task_info <Task ID>

    例：

    vim-cmd vimsvc/task_info haTask-109-vim.vm.Snapshot.remove-92744898

    これにより、タスクを特定するのに役立つ情報（短縮形）が出力されます。

    task = 'vim.Task:haTask-109-vim.vm.Snapshot.remove-92744898',
    state = "running",
    cancelled = false,
    cancelable = true,
    progress = 75,
    startTime = "2014-12-06T03:47:26.30322Z",
     
  10. Restart the Management Agent on this ESXi host and clear the services/tasks. For more information, see Restart Management Agents on ESX and ESXi Hosts (1037058).

    Note: To avoid unnecessary failovers, disable HA Host Monitoring first.

    After a few seconds, run the command get.tasklist (see above) again. You should see an empty output. You should be able to integrate/power on.
In some cases, this process requires additional cleanup work.
  1. Run the first command again to verify that the virtual machine is still registered (or that the status is invalid in vCenter).

    vim-cmd vmsvc/getallvms | grep <ID>

    Skipping invalid VM '109'Note

    : If you are not getting invalid VM output, skip the rest of this section.
     
  2. This output indicates that there is a conflict. The virtual machine might still be running, but no ID has been assigned. Run the following command:

    esxcli vm process list | grep -i <vm-name> -B5

    output shows a list of virtual machines. Make a note of the world ID of this virtual machine.

    esxcli vm process kill -t force -w <World ID>

    Note: This kills the process of the virtual machine (hard shutdown). Please use it at your own risk. Also, if the virtual machine is responding, RDP to the virtual machine and shut it down at the guest operating system level. If you run the esxcli vm process list command above again (after a few seconds), the output should be empty. Remove the virtual machine from the vCenter inventory and add it back to it.
    You should be able to integrate/power on.
     
  3. If you are still unable to consolidate/power on the virtual machine, file a support request with VMware. For more information, see How to file a support request in Broadcom Portal (2079666).

Troubleshooting a virtual disk that is locked on an NFS volume

Locking issues on NFS datastores are different from locking issues on VMFS datastores. NFS does not provide block-level access and avoids SCSI locking, so the locking mechanism is different. NFS locking is implemented by creating a lock file on the NFS server. When you browse an NFS datastore to view hidden files, the . lck-#### A large number of files are displayed. Because of this locking mechanism, you cannot use the same command-line tool to determine what is locking.

Power off the backup appliance and the virtual machines, as well as any other virtual machines that might access the virtual disk.

1. Find your locks
 
  1. Connect as root to the ESXi host to which the virtual machine is registered, using SSH, and browse to the datastore.
  2. Run the following command to create the . lck-#### Show hidden files.
# ls -lha
If the virtual machine is powered off and there is no other access to the virtual disk, the . The lck-#### file is missing.

2. Information about the lockIf


you have an .lck-#### file, you can get more information about its origin by running the following command and looking at its contents:
# strings .lck - #### (replace with the correct file name) On

ESXi 5.5 and later, run the following command:

# hexdump -C .lck - #### (replace with the correct file name)
This gives you the hostname of the lock owner. Example: sxhost2.domain.local

3.


You can then use the rm command (as long as the virtual machine is powered off) to delete this file.
# rm .lck-#### (replace with the correct file name)
Run the same ls -lha command after a few seconds to see if the lock has been rewritten.

If so, find out which virtual machine is owned by this ESXi host and find the virtual machine that is causing this problem (usually an ISO from NFS that is mounted as a backup appliance or CD/DVD).
If not, you should be able to integrate/power on.

D. The problem is not caused by the .lck-#### file, but by a general connection issueThis


article does not consider the issues that occur due to general NFS connectivity issues. 

When you browse (via SSH or a datastore browser) to a virtual machine directory that causes

a disk module premature error, you see a file name similar to the following:
000002-S001.VMDK
000002-S002.VMDK
000002-S003.VMDK
[...]

1. Information of


2GBSPARSEA sparse disk with an extended size of up to 2 GB. Disks in this format can be used with other VMware products, but to power on a sparse disk on an ESXi host, you must re-import the disk in a compatible format (such as Thick or Thin) using the vmkfstools command.

2. Cause:


This format only occurs when you use VMware Converter (P2V) and select the wrong disk type. Fix: Convert again and use the correct format (Thin, Thick, etc.).

3. Workaround/Fix:


Clone Disk in Supported Formats
  1. Log in to the ESXi Shell, load the multi-extent module, and run the following commands:

    # vmkload_mod multiextent

    should be able to integrate/power on temporarily.
     
  2. Clone the source ource.vmdk of the hosted disk to new.vmdk by running the following command:

    # vmkfstools -i source.vmdk new.vmdk -d zeroedthick|eagerzereodthick|thin
     
  3. After successful cloning, run the following command to delete the hosted disk test1.vmdk:

    # vmkfstools -U source.vmdk
     
  4. Rename the cloned vmfs type disk new.vmdk to source.vmdk.

    # vmkfstools -E new.vmdk nsource.vmdk You

    should be able to integrate/power on.
     
  5. Unload the multi-extent module with the following command:

    # vmkload_mod -u multiextent


Additional Information

ESXi 6.x 以降は NFSv4.1 互換を使用して、他のメリットの中でも Kerberos 5 認証をサポートしています。

vSphere 6.x 以降の場合、VMware では、別の ESXi ホスト上の NFS データストアをマウントする場合に以下をお勧めしています。
  • ESXi ホストで NFS プロトコルのバージョンを混在させない。
  • Network Attached Storage (NAS) を構成して 1 つのプロトコル バージョンのみを使用する。
  • NFS に接続する全 ESXi ホストで IPv4 または IPv6 を混在させない。

Troubleshooting issues resulting from locked virtual disks

Powered by Wolken Software