Using thin provisioned disks with virtual machines

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article provides information on thin provisioned disks within VMware ESXi/ESX.

Resolution

What is a thin provisioned disk?

When creating a virtual disk file, by default VMware ESXi/ESX uses a thick type of virtual disk. The thick disk pre-allocates all of the space specified during the creation of the disk. For example, if you create a 10 megabyte disk, all 10 megabytes are pre-allocated for that virtual disk.

In contrast, a thin virtual disk does not pre-allocate the space. Blocks in the VMDK file are not allocated and backed by physical storage until they are written during the normal course of operation. Reading an unallocated block returns zeroes, but the block is not backed with physical storage until it is written.

Considerations

When implementing thin provisioning in VMware environment:

Thin provisioned disks can grow to the full size specified at the time of virtual disk creation, but do not shrink. After allocating blocks, blocks cannot be un-allocated.
By implementing thin provisioned disks, you are able to over-allocate storage. If storage is over-allocated, thin virtual disks can grow to fill an entire datastore if left unchecked.
With NFS datastores, the provisioned disk format is thin provisioned by default and cannot be changed. With vSphere 5.0, you can specify the provisioned format. For example, you can specify thick provisioning if the storage array or filer supports it through VAAI.
VMware ESX 3.x is not aware of thin provisioning when reporting disk space usage using VMware Infrastructure Client and VirtualCenter.
VMware ESX 4.x is aware of thin provisioning in the form of the storage views plug-in for vCenter Server.
When a thin provisioned disk grows, the VMFS metadata must be locked to make changes to a file. The ESXi/ESX host must make a SCSI reservation to make changes. For more information about SCSI reservations, see Analyzing SCSI Reservation conflicts on VMware Infrastructure 3.x, vSphere 4.x, vSphere 5.x and vSphere 6.0
If a guest operating system needs to make use of a virtual disk, the guest operating system must first partition and format the disk to a file system it can recognize. Depending on the type of format selected within the guest operating system, the format may cause the thin provisioned disk to grow to full size.

For example, if you present a thin provisioned disk to a Microsoft Windows operating system and format the disk, unless you explicitly select the Quick Format option, the Microsoft Windows format tool writes information to all of the sectors on the disk, which in turn inflates the thin provisioned disk to full size.

How is thin provisioning used in VMware ESXi/ESX?

In ESX 3.x and VirtualCenter 2.x, the use of thin provisioning should be limited to templates. When deploying a virtual machine that used for production, VMware recommends using the Deploy a VM from template option instead of the Convert this template to VM option. This ensures that the virtual machine which is deployed using thick virtual disks. In ESX 3.x, the use of thin virtual disks in production may result in performance degradation since disk space for the thin disks is allocated as the disks grow, and the allocated disk space is likely not contiguous.

In ESX/ESXi 4.x and later versions, thin virtual disks are fully supported in production environments. You can specify whether you want create a thin or thick disk when creating virtual machines or adding disks to existing virtual machines.

Verifying how much space a thin provisioned virtual disk file is using

These steps demonstrate the differences between a normal and a thin disk using the service console. Similar steps can be used to verify the space usage of a pre-existing thin provisioned virtual disk file:

Note: These steps make use of the du command, which is only available in the ESX classic service console.

Create a normal virtual disk file by running the command:

vmkfstools -c 10M normal.vmdk
Run the command:

ls -alh normal*.vmdk

The output is similar to:

-rw------- 1 root root 10M normal-flat.vmdk -rw------- 1 root root 367 normal.vmdk

Note the file sizes. In the example above, the ls utility reports that the normal-flat.vmdk file is 10 MB in size as specified in step 1.
Run the command:

du -h normal*.vmdk

The output is similar to:

10M normal-flat.vmdk 64K normal.vmdk

Note that the du (disk usage) utility also reports that the normal-flat.vmdk file is 10 MB in size.
Remove the files by running the command:

vmkfstools -U normal.vmdk
Create a thin virtual disk file by running the command:

vmkfstools -c 10M -d thin thin.vmdk
Run the command:

ls -alh thin*.vmdk

The output is similar to:

-rw------- 1 root root 10M thin-flat.vmdk -rw------- 1 root root 391 thin.vmdk

Note the file sizes. In the example above, the ls utility reports that the thin-flat.vmdk file is 10 MB in size as specified in step 7.
Run the command:

du -h thin*.vmdk

The output is similar to:

2.0M thin-flat.vmdk 64K thin.vmdk

Note: The du (disk usage) utility reports that the thin-flat.vmdk file is only 2 MB in size.
Remove the files by running the command:

vmkfstools -U thin.vmdk

Thin provisioning and storage devices

Depending on the type of storage device you are using to house a virtual disk file, it may employ a similar mechanism on the back end to maximize storage. When using NFS to store your virtual disks, you may observe similar disk usage to that of a thin provisioned virtual disk. The lower disk usage can be observed even when explicitly creating a normal or thick virtual disk.

These steps demonstrate thin provisioning on an NFS datastore:

Note: These steps make use of the du command, which is only available in the ESX classic service console.

Log into the ESX host via the Service Console as root.
Navigate to an NFS datastore.
Create a normal virtual disk file by running the command:

vmkfstools -c 10M thick.vmdk
Run the command:

ls -alh thick*.vmdk

The output is similar to:

-rw------- 1 root root 10M thick-flat.vmdk -rw------- 1 root root 367 thick.vmdk

Note the file sizes. In the example above, the ls utility reports that the thick-flat.vmdk file is 10 MB in size as specified in step 3.
Run the command:

du -h thick*.vmdk

The output is similar to:

136K thick-flat.vmdk 8.0K thick.vmdk

Note: The du (disk usage) utility reports that the thick-flat.vmdk file is only 136 KB in size.
Remove the files by running the command:

vmkfstools -U thick.vmdk

Note: When using thin provisioned disks with Microsoft operating systems such as Windows 2008 or Windows 7, if you perform a Full Format of the disk, this will inflate the thin provisioned disk to the full size. To avoid this situation, select the option to perform a Quick Format.