ESX/ESXi host locks a series of on-disk resource metadata structures delaying operations from other hosts

Products

• VMware

Issue/Introduction

Symptoms:

• ESX/ESXi host locks a series of on-disk resource metadata structures, resulting in delays for operations from another host.
• This article only applies when the "type 10c00002" is seen in the error message.
• In the vmkernel.log file (located at /var/log) of the hosts that are being delayed, you see entries similar to:
  
  DLX: 3394: vol 'datastore-name': [Req mode 1] Checking liveness of [type 10c00002 offset 23714816 v 8, hb offset 3903488
  gen 53, mode 1, owner 50a3da5a-a3b687f2-9b6a-842b2b0c3b60 mtime 255486 nHld 0 nOvf 0]
  DLX: 3901: vol 'datastore-name': [Req mode: 1] Not free; Lock [type 10c00002 offset 23714816 v 8, hb offset 3903488
  gen 53, mode 1, owner 50a3da5a-a3b687f2-9b6a-842b2b0c3b60 mtime 255486 nHld 0 nOvf 0]
  

Cause

During file related activities on a VMFS datastore, the ESX/ESXi host acquires a lock on a series of on-disk resource metadata structures to either allocate file blocks to a file or free file blocks while deleting the file from VMFS datastore. After it completes the resource metadata structure related operations, the ESX/ESXi host continues to retain the lock for a short period of time in case the host gets further requests for allocations. In an ESX/ESXi cluster where multiple hosts are actively working with the same VMFS datastore, this lock from one host can delay operations from another.

Under most conditions the delay is unnoticed operationally. However, the chance of this delay becoming apparent to user level operations increases under these conditions:

• Large number of hosts (>10) in a cluster attempting to allocate files on VMFS datastores with low % free space (< 10%).</div>
• Power-off or power-on operations on many virtual machines in a short period of time.
• vMotion of virtual machines that involves VMFS datastores with low % free space (< 10%).</div>
• High activity on disk trees that contain a lot of linked clones.
• High rate of growth with thin disk provisioning.
  

Resolution

The issue is resolved in:

• ESXi 5.0 patch. For more information, see VMware ESXi 5.0, Patch ESXi500-201308401-BG: Updates esx-base (2053133).
• ESXi 5.1 patch. For more information see VMware ESXi 5.1, Patch ESXi510-201307401-BG: Updates esx-base and misc-drivers (2052144)
• ESXi 5.5 patch. For more information, see VMware ESXi 5.5, Patch Release ESXi550-201312001 (2063795).
  

You can download the latest release from the VMware Download Center.

To work around this issue when you are unable to apply a patch:

• Run rescans from individual hosts one at a time using the vSphere Client or console. For more information, see Performing a rescan of the storage on an ESXi/ESX host (1003988).
• Disable automatic rescan from vCenter Server. For more information, see How to disable automatic rescan of HBAs initiated by vCenter (1016873).
• Reduce the number of LUNs/datastores that every host is seeing. For each host, only present the LUNs that the cluster hosts have virtual machines or RDMs on. Unpresent all the unused LUNs from the cluster.

Additional Information

How to disable automatic rescan of HBAs initiated by vCenter Server
Cluster wide storage rescan from vCenter Server causes ESXi/ESX hosts and virtual machines to become unresponsive
VMware ESXi 5.1, Patch ESXi510-201307401-BG: Updates esx-base and misc-drivers
VMware ESXi 5.5, Patch Release ESXi550-201312001
ESX/ESXi ホストが一連のオンディスク リソース メタデータ構造をロックし、他のホストからの操作が遅れる