VMFS deleted blocks reclamation on thin-provisioned LUNs

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides steps to reclaim unused storage blocks on a VMFS datastore for a thin-provisioned LUN. 

 

vSphere 5.5 introduced a new command in the esxcli namespace that allows deleted blocks to be reclaimed on thin provisioned LUNs that support the VAAI UNMAP primitive.

Execute the command the command without any maintenance window, and the reclaim mechanism has been enhanced:

• Reclaim disk space size can be specified in blocks instead of a percentage value to make it more intuitive to calculate.
• Dead disk space is reclaimed in increments instead of all at once to avoid possible performance issues.

With the introduction of 62 TB VMDKs, UNMAP can now handle much larger dead space areas. However, UNMAP operations are still manual. This means Storage vMotion or Snapshot Consolidation tasks, for example, on VMFS do not automatically reclaim disk space on the array LUN.

Environment

VMware vSphere ESXi 8.X

 vSphere 7.x and

 

VMFS-6

Resolution

There are two ways in which the disk space reclamation tasks could be invoked:

• Using the esxcli command
• Using the vmkfstools command

1. Verify VAAI UNMAP Support Before attempting reclamation, confirm the LUN supports the Delete Status primitive.

1. Open an SSH session to the ESXi host.
2. Run the following command:
   bash

    

   esxcli storage core device vaai status get -d naa.##############

3. Ensure the output shows Delete Status: supported. If it shows unsupported, the physical array does not support SCSI UNMAP for this device.

2. Execute Manual UNMAP (VMFS-5) Manual reclamation is required for VMFS-5 or if automatic reclamation is disabled on VMFS-6.

1. Identify the volume label or UUID of the datastore.
2. Execute the reclamation command:
   bash

    

   esxcli storage vmfs unmap --volume-label=volume_label --reclaim-unit=200

   Note: The default --reclaim-unit is 200 blocks. Consult storage vendors (e.g., Pure Storage, NetApp) for optimized values.

3. Monitor Progress To track the reclamation progress in real-time:

1. Run esxtop.
2. Press u to switch to the disk device view.
3. Press f and select O to enable VAAI statistics.
4. Monitor the DELETE and MBDEL/s counters. These indicate the number of UNMAP commands issued and the throughput of the reclamation.

Using the esxcli command

To reclaim unused storage blocks on a VMFS datastore for a thin-provisioned device, run this command:


esxcli storage vmfs unmap -l MyDatastore

or

esxcli storage vmfs unmap -u ########-########-####-############

Notes:

• The default value of 200 for the -n number or --reclaim-unit=number argument is appropriate in most environments, but some array vendors may suggest a larger or smaller value depending on how the array handles the SCSI UNMAP command.
• This command creates temporary hidden files at the top level of the datastore, with names using the .asyncUnmapFile pattern. By default, the space reservation for the temporary files depends on the block size of the underlying VMFS file system (the default is --reclaim-unit=200):
   
  • 200 MB for 1 MB block VMFS3 / VMFS5
  • 800 MB for 4 MB block VMFS3
  • 1600 MB for 8 MB block VMFS3
     
• Depending on the use case, you can select a different --reclaim-unit value, for example, if the reserved size is considered to be too large or if there is a danger that the UNMAP primitive may not be completed in a timely manner when offloaded to an array. VMware recommends that you consult with your storage array providers on the best value or best practices when manually defining a --reclaim-unit value.
  
  
• If the UNMAP operation is interrupted, a temporary file may be left on the root of a VMFS datastore. However, when you run the command against the datastore again, the file is deleted when the command completes successfully. The .asyncUnmapFile will never grow beyond the --reclaim-unit size.
  
  

VMFS-5 requires UNMAP, whereas VMFS-6 supports automatic reclamation (https://knowledge.broadcom.com/external/article/316475/migrating-vmfs-5-datastore-to-vmfs-6-dat.html).

• The esxcli storage vmfs unmap command does not utilize the Zero Page Reclaim function. During the UNMAP process, ESXi sends SCSI UNMAP commands to the storage array, notifying it of free blocks that can be deallocated. The array must then execute the necessary zero-page/zero-block operations internally to complete the reclamation process. If the zero-page operations fail or are inefficient at the array level, an alternative is to use the FStrim command within the guest operating system. However, the exact end-to-end workflow and support for this method should be confirmed with the SAN array vendor.

Using the vmkfstools command

Alternatively, you could use vmkfstools with the switch -y to reclaim unused storage blocks. Historically, this command could only be used to reclaim only a certain percentage of space, but now, this has been tweaked to accommodate blocks.

The use case would be as follows:

vmkfstools -y  /vmfsPath
[--reclaimBlocksUnit #blocks]Fsimply

Example : 

vmkfstools --reclaimBlocksUnit 200 -y /vmfs/volumes/vmfsPath

Notes:

• You could leverage the esxtop command to monitor the progress of the unmap process. For more information, see Monitoring VMware vSphere Storage APIs – Array Integration using esxtop and maintaining the status as enabled.
• The UNMAP operation may finish without doing anything or fail if the volume partition table and/or the block alignment is incorrect, due to upgrading a VMFS3 files system or repartitioning the volume with a 3rd party tool. For more information, see Thin Provisioning Block Space Reclamation (VAAI UNMAP) does not work.
• If the UNMAP operation fails and you see an error about "resource busy", if possible evacuate the datastore and try the unmap command again. If still fails or you see an error about "locked files", please check:
  • Removing VMFS Volume locks on datastores
  • Device or resource busy/ file is locked error when making changes to a file on an ESXi datastore
  • Investigating virtual machine file locks on ESXi hosts

Note: 

Deleting a VMFS datastore from vSphere will not automatically reclaim space back into your backend storage pool, regardless of whether the LUN is thin or thick provisioned. When you Delete Datastore on ESXi, vSphere merely destroys the VMFS filesystem structure and clears the partition table. It does not send a wholesale SCSI UNMAP command to wipe the entire raw device. To the physical storage array, the LUN still exists, and any blocks that were previously written to remain occupied.The storage administrator must log into the SAN array management console and completely delete the LUN.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Thin Provisioning Block Space Reclamation (VAAI UNMAP) does not work

 

• Migration: To automate space reclamation, migrate workloads from VMFS-5 to a newly created VMFS-6 datastore. In-place upgrades are not supported.
• Contact Support: If the UNMAP command fails with "resource busy" or "locked files," refer to the troubleshooting guides or Contact Broadcom Support.