Understanding large-scale sequential write performance in VMDKs residing on VMFS datastores
search cancel

Understanding large-scale sequential write performance in VMDKs residing on VMFS datastores


Article ID: 343258


Updated On:


VMware vSphere ESXi


This article describes the distinctions between the provisioning methods and their consequences on persistent write performance. Large-scale write IO performance may improve or degrade depending on provisioning scheme.

When a guest OS drives large amount of sequential write IO to a disk, write performance may differ based on the provisioning type used for the virtual disk. VMware supports three provisioning types:
  1. Thin Provisioned
  2. Thick Provisioned
  3. Eager-zeroed Thick Provisioned
These three provisioning methods will have different performance consequences depending on workload and IO distribution. The largest distinctions are seen in large-scale write IO performance. Depending on provisioning scheme, you may observe a significant performance differential during large-scale sequential write IO operations. In certain circumstances, performance may be significantly lower/degraded in one provisioning scheme versus others. Understanding the distinction between the provisioning schemes for sequential write IO operations will clarify potential performance improvement or degradation when running large-scale sequential write IO operations.


VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0


The primary distinction between the three provisioning types surrounds preparation of net new blocks when they are consumed for the first time by a VM. The three provisioning schemes have different default, day zero behavior:
VMDK Provisioning TypeBlocks allocated on creation?Blocks zeroed on creation?
Eager-zeroed ThickYY
In this context, "allocated" means that the space is marked as in-use by the VMFS filesystem. This guarantees that the blocks promised to the VMDK will be available, and prevents overprovisioning of storage resources. "Zeroed" means that the blocks are all prepared, erased and ready for consumption by the VM. These distinctions can cause difference in write IO performance, as different VMFS filesystem-level steps are required depending on provisioning type. When a VM issues a write to a virtual disk, the block being written may need to go through a preparation phase depending on allocation scheme. All preparation in VMFS is done in 1MiB increments.* This preparation only occurs when a new block is touched for the first time.
  1. New blocks on thin-provisioned VMDKs
    • The space needed to satisfy the new block is allocated on the VMFS filesystem.
    • The new block is zeroed by VMFS to ensure consistency.
    • The new block is ready, and the VM's write is processed. The payload is written to the new block.
  2. New blocks on thick-provisioned VMDKs
    • The new block is zeroed by VMFS to ensure consistency
    • The new block is ready, and the VM's write is processed. The payload is written to the new block.
  3. New blocks on eager-zeroed thick-provisioned VMDKs
    • Not applicable. All blocks are pre-allocated and pre-zeroed when eager-zeroing is used. The payload is written immediately, as there are no net-new blocks.
The preparation steps associated with thin and regular thick-provisioned VMDKs do add some overhead to write operations. During normal production, these are not meaningful and performance is similar across all provisioning types. During large-scale consumption of new blocks, however, performance distinctions may be observable to the end user. As the payload cannot be written until the block is prepared, the payload must wait until preparation is complete before it can be written. When large write operations are initiated by a guest OS, this will involve a large volume of block preparation activities and could result in observably lower performance on thin or regular thick-provisioned disks when performance is compared to eager-zeroed disks.

The degree of any observable performance differential will vary based on numerous factors including:
  • Overall load level within the compute environment
  • Load on the storage array
  • The storage array's handling of zero data
  • Other factors.
The overheads involved in thin and normal thick-provisioned VMDKs are modest, though at high volume they can become apparent due to these or numerous other factors. The primary external factor (beyond ESXi and VMFS behaviors) in write performance are specific storage array behaviors. Various storage platforms handle zero data differently (such as transparent page compression, thin volume/LUN zeroing behaviors, etc.) and this can cause changes or degradation in end-user performance within VMs. For additional information regarding best practices for provisioning schemes, array-specific handling of zero data, thin volume/LUN preparation or other topics, VMware recommends engagement with the applicable storage vendor for specific guidance.

In general, thin provisioned VMDKs are useful as they enable easier space management and result in less wasted space through vSphere advanced space-reclamation technologies both at the datastore level and within many guest operating systems when thin provisioning is used. Eagerzeroed thick provisioning is typically recommended only for highly latency-sensitive applications or where the allocation/zeroing process overhead should be avoided. This recommendation also may vary depending on array manufacturer recommendations.

* Since VMFS5, all internal VMFS block sizes are 1MiB. In VMFS3, the block size was 1MiB by default, or user-selectable at any doubling between 1MiB and 8MiB.