Reclaiming VMFS deleted blocks on Thin Provisioned LUNs

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps to reclaim unused storage blocks on a VMFS datastore for a thin-provisioned LUN.

vSphere 5.5 introduced a new command in the esxcli namespace that allows deleted blocks to be reclaimed on thin provisioned LUNs that support the VAAI UNMAP primitive.

You can run the command without any maintenance window, and the reclaim mechanism has been enhanced:

Reclaim disk space size can be specified in blocks instead of a percentage value to make it more intuitive to calculate.
Dead disk space is reclaimed in increments instead of all at once to avoid possible performance issues.

With the introduction of 62 TB VMDKs, UNMAP can now handle much larger dead space areas. However, UNMAP operations are still manual. This means Storage vMotion or Snapshot Consolidation tasks, for example, on VMFS do not automatically reclaim disk space on the array LUN.

Environment

VMware vSphere ESXi 6.0

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.#

VMware vSphere ESXi 8.#

Resolution

Note: If you are using datastores as VMFS-6 in ESXi 6.5 or above, this article is not applicable.

There are two ways in which the disk space reclamation tasks could be invoked:

Using the esxcli command
Using the vmkfstools command

To confirm if SCSI UNMAP is supported on a LUN, open an SSH session to an ESXi host and run this command:

esxcli storage core device vaai status get -d naa.##############

You see output similar to:

naa.<ID>
VAAI Plugin Name: VMW_VAAIP_NETAPP
ATS Status: supported
Clone Status: supported
Zero Status: supported
Delete Status: supported

Using the esxcli command

To reclaim unused storage blocks on a VMFS datastore for a thin-provisioned device, run this command:

esxcli storage vmfs unmap --volume-label=volume_label|--volume-uuid=volume_uuid --reclaim-unit=number

i.e. specify the volume by label or uuid:
-l|--volume-label=volume_label
The label of the VMFS volume to UNMAP.

-u|--volume-uuid=volume_uuid
The UUID of the VMFS volume to UNMAP.

optionally specify:
-n|--reclaim-unit=number
The number of VMFS blocks to UNMAP per iteration. If it is not specified, the command uses a default value of 200.

For example, for a VMFS volume named MyDatastore with UUID of 509a9f1f-########-####-########780e, run this command:

esxcli storage vmfs unmap -l MyDatastore

or

esxcli storage vmfs unmap -u 509a9f1f-########-####-########780e

Notes:

The default value of 200 for the -n number or --reclaim-unit=number argument is appropriate in most environments, but some array vendors may suggest a larger or smaller value depending on how the array handles the SCSI UNMAP command.
This command creates temporary hidden files at the top level of the datastore, with names using the .asyncUnmapFile pattern. By default, the space reservation for the temporary files depends on the block size of the underlying VMFS file system (the default is --reclaim-unit=200):
- 200 MB for 1 MB block VMFS3 / VMFS5
- 800 MB for 4 MB block VMFS3
- 1600 MB for 8 MB block VMFS3
Depending on the use case, you can select a different --reclaim-unit value, for example, if the reserved size is considered to be too large or if there is a danger that the UNMAP primitive may not be completed in a timely manner when offloaded to an array. VMware recommends that you consult with your storage array providers on the best value or best practices when manually defining a --reclaim-unit value.
If the UNMAP operation is interrupted, a temporary file may be left on the root of a VMFS datastore. However, when you run the command against the datastore again, the file is deleted when the command completes successfully. The .asyncUnmapFile will never grow beyond the --reclaim-unit size.

Using the vmkfstools command

Alternatively, you could use vmkfstools with the switch -y to reclaim unused storage blocks. Historically, this command could only be used to reclaim only a certain percentage of space, but now, this has been tweaked to accommodate blocks.

The use case would be as follows:

vmkfstools -y --reclaimBlocks /vmfsPath
[--reclaimBlocksUnit #blocks]

Example :

vmkfstools -y --reclaimBlocks 200 /vmfs/devices/disks/naa.<ID>

Notes:

You could leverage the esxtop command to monitor the progress of the unmap process. For more information, see Monitoring VMware vSphere Storage APIs – Array Integration using esxtop and maintaining the status as enabled (broadcom.com).
The UNMAP operation may finish without doing anything or fail if the volume partition table and/or the block alignment is incorrect, due to upgrading a VMFS3 files system or repartitioning the volume with a 3rd party tool. For more information, see Thin Provisioning Block Space Reclamation (VAAI UNMAP) does not work (broadcom.com).
If the UNMAP operation fails and you see an error about "resource busy", if possible please evacuate the datastore and try the unmap command again. If still fails or you see an error about "locked files", please check:
Device or resource busy/ file is locked error when making changes to a file on an ESXi datastore (broadcom.com)
- Investigating virtual machine file locks on ESXi hosts (broadcom.com).

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ (broadcom.com)

Thin Provisioning Block Space Reclamation (VAAI UNMAP) does not work (broadcom.com)