Thin Provisioning Block Space Reclamation (VAAI UNMAP) does not work

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
When using thin provisioned SAN LUN(s) with sufficient space on the datastore and the SAN array supplying the shared storage supports the vStorage APIs for Array Integration (VAAI) primitive UNMAP:

• Issuing the UNMAP command (using the vmkfstools -y command) to the array succeeds, but the operation completes in few seconds, regardless of the amount of space that can be reclaimed.
   
• When you check the results of the UNMAP command, you see that nothing has changed on the SAN array and the usage of the thin provisioned LUN has not decreased.
   
• Running the esxcli storage vmfs unmap -l datastore command in vSphere 5.5 to UNMAP, you see an error similar to
  
  FSDisk: 262: Issue of delete blocks failed [sync:1] numDescs 1 Address is not aligned on required boundary 0x412fc3750540

Environment

VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.5
VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Embedded
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 6.0
VMware ESXi 3.5.x Installable
VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.1

Cause

This issue occurs due to a misaligned partition table. For VAAI to function properly, some arrays require specific alignment of the underlying partition.

The partition table may be misaligned due to one of these reasons::

• The volume is not native VMFS-5 (it is converted from VMFS-3)
• The partition table of the LUN is created manually

Note: If the VMFS datastores are created using the vSphere Client or vCenter Server, it is likely that all requests are aligned. If a LUN is incorrectly partitioned using low-level tools such as fdisk or partedUtil, the operation might revert to use the software Data Mover.

For more information, see VMware vSphere Storage APIs – Array Integration (VAAI) documentation.

Resolution

The VAAI primitive UNMAP only works on SAN arrays where the partition offset is a multiple of 1 MB.

To resolve this issue, create and use a VMFS-5 datastore:

1. Migrate any files off the upgraded VMFS-5 datastore using Storage vMotion or cloning.
2. Delete the now empty upgraded datastore.
3. Confirm that the LUN previously housing the problem datastore is now available for use in Add Storage.
4. Create a native VMFS-5 datastore on the LUN.

To confirm that VAAI UNMAP is now working correctly:

1. Create a test virtual machine on the new native VMFS-5 datastore.
2. Delete the test virtual machine.
3. Retry an UNMAP operation using the vmkfstools -y command. For ESXi 5.5 host, see Using ESXCLI in vSphere 5.5 to reclaim VMFS deleted blocks on thin-provisioned LUNs (2057513).
   
   You now see the space being reclaimed correctly on the thin provisioned SAN LUN.

Note:

• The vmkfstools -y command works regardless of the status of the EnableBlockDelete variable. EnableBlockDelete is used to disable/enable the automated space reclamation. For more information, see Disabling VAAI Thin Provisioning Block Space Reclamation (UNMAP) in ESXi 5.0 (2007427).
• The vmkfstools -y command only works in vSphere 5.0 U1 or later and vSphere 5.1. For more information on UNMAP, see Using vmkfstools to reclaim VMFS deleted blocks on thin-provisioned LUNs (2014849).

Additional Information

Native VMFS-5 volumes have a GPT partition table. To determine if a LUN has a GPT partition table:
 

1. To check the partition type, run this command:
   
   # esxcli storage core device partition showguid
   
   Example output:
   
   Device Partition Layout GUID
-------------------------------------------------------------
naa.6090a078607099871de8a463b767816c 0 MBR N/A
naa.6090a078607099871de8a463b767816c 1 MBR N/A
naa.6090a0c820f0398e9108b5a86a01c017 0 GPT 00000000000000000000000000000000
naa.6090a0c820f0398e9108b5a86a01c017 1 GPT aa31e02a400f11db9590000c2911d1b8
   
   In this example, there are two LUNs:
    
   • LUN 1: VMFS-5 (upgraded from VMFS-3) MBR partition
     Unique Identifier: naa.6090a078607099871de8a463b767816c
      
   • LUN 2: VMFS-5 (native VMFS-5) GPT partition
     Unique Identifier: naa.6090a0c820f0398e9108b5a86a01c017
2. To check the offset of the partitions, run this command:
   
   # esxcli storage core device partition list
   
   Example output:
   
   Device Partition Start Sector End Sector Type Size
-------------------------------------------------------------------------------------
naa.6090a078607099871de8a463b767816c 0 0 3221237760 0 1649273733120
naa.6090a078607099871de8a463b767816c 1 128 3221225280 fb 1649267277824
naa.6090a0c820f0398e9108b5a86a01c017 0 0 3221237760 0 1649273733120
naa.6090a0c820f0398e9108b5a86a01c017 1 2048 3221237727 fb 1649272667648
   
   On LUN 1, the first partition has an offset of 128 sectors, which is 64 KB (128*512 bytes), while the first partition on LUN 2 (native VMFS-5 datastore with a GPT partition) has an offset of 2048 sectors, which is 1 MB (2048*512 bytes).

Disabling the VAAI functionality in ESXi/ESX
Disabling VAAI Thin Provisioning Block Space Reclamation (UNMAP) in ESXi 5.0
How to reclaim VMFS deleted blocks on thin-provisioned LUNs
Using the esxcli storage vmfs unmap command to reclaim VMFS deleted blocks on thin-provisioned LUNs
"Issue of delete blocks failed" error is displayed in vmkernel.log file


KB 000051613