Monitor automatic unmap I/O issued by ESXi

Products

VMware vSphere ESXi

Issue/Introduction

This article explains the automatic SCSI unmap operation performed by ESXi for space reclamation.

With VMFS-6, space reclamation is now an automatic, asynchronous process. This means that the space will not be reclaimed instantly but gradually over time. As a result, you may need to monitor the process to track progress.

In this article explains the automatic unmap for VMFS-6:

How to check the automatic unmap I/O operations issued by an ESXi host for a VMFS-6 datastore or volume.
How to monitor unmap activity using esxtop and vsish commands .

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Resolution

How to check the automatic unmap I/O per VMFS6 datastore/volume issued by an ESXi host.and check if automatic unmap I/O has been issued by an ESXi host.

On an SSH session to an ESXi host, access the VSI shell prompt by running:

# vsish

Then navigate as follows:

/> cd /vmkModules/vmfs3/auto_unmap/volumes/

To list the VMFS6 datastores/volumes for which unmap is being issued, run:

/> ls

To view the automatic unmap stats for a specific datastore:

/> get /vmkModules/vmfs3/auto_unmap/volumes/<datastoreName>/properties

Alternatively, directly run:

vsish -e get /vmkModules/vmfs3/auto_unmap/volumes/<datastoreName>/properties

Sample output:

Volume specific unmap information {
Volume Name :<datastoreName>
Volume: Num SchedQs :1
FS Major Version :24
Metadata Alignment :4096
Allocation Unit/Blocksize :1048576
Unmap granularity in File :1048576
Volume: Unmap IOs :1446
Volume: Unmapped blocks :85135
Volume: Num wait cycles :1
Volume: Num from scanning :3948
Volume: Num from heap pool :391
Volume: Total num cycles :961963
Unmaps processed in last minute:0
Volume: Unmap SchedQ IOs :1446
}

How to monitor the Umap activity via ESX top.

SSH into the ESXi host, type “esxtop” then enter and then type “u” then “f” then select VAAI statistics with “o”.

On the far right, you will see delete stats, aka UNMAP:

DELETE is a counter for UNMAP commands issued to that device, from that host since the last reboot.
DELETE_F is a counter for failed UNMAP commands issued to that device, from that host since the last reboot.
Note that this isn’t like how many times someone ran the UNMAP command. It is literally a count of UNMAP I/Os.
A single UNMAP operation could include thousands of UNMAP commands.
The last column is not a counter, but instead a indicator of the throughput of any UNMAP operations going on right now from that host to that device. So if no UNMAP operations are running right now, it will be zero.

Note :

Both vsish and esxtop counters are reset when the ESXi host reboots.
"vsish" counters are also reset if a datastore is no longer being monitored. For example, if a running VM is on a datastore, that datastore is actively monitored and UNMAPPED by the host. However, if the VM is powered off or moved to another host, and there are no VM running on the datastore for that host, the datastore becomes unmonitored, causing the counters to reset to 0.

Additional Information

On VMFS datastores, storage space is released in several ways:

When a VM is deleted.
When a VM is migrated using vSphere Storage vMotion.
When a snapshot is deleted or consolidated.

Within the Guest OS, storage space is freed when files are deleted. Most modern Guest OSes automatically send these frees as "Unmaps" to the VMFS datastore. However, not all Guest OS support automatic Unmaps.

In VMFS6, VMware has enhanced this process by supporting the automatic tracking of free blocks, which is then used for periodic asynchronous Unmap operations. This feature is available only for datastores created on Thin Provisioned Storage LUNs.

Asynchronous automatic Unmap processing provides several key benefits:

Consistent Load on Storage: Unmap requests are sent at a steady rate, preventing sudden, burst loads on storage arrays.
Efficient Batch Processing: Freed storage blocks are unmapped in batches, improving efficiency.
Improved I/O Performance: Unmap processing and truncate I/O operations are decoupled, ensuring that IO performance remains unaffected.

Non-zero values for Volume: Unmap IOs and Volume: Unmapped Blocks indicate that automatic unmap I/Os have been issue by a host for a given VMFS6 volume since last reboot.

Check the output at intervals (e.g. daily) for incrementing values of Volume: Unmap IOs and Volume: Unmapped Blocks to confirm ongoing issue of automatic unmap I/Os to the volume on the host.

Unmap metrics can also be monitored via the VAAI stats on esxtop - see Monitoring VMware vSphere Storage APIs – Array Integration using esxtop and maintaining the status as enabled. Note, however, that esxtop unmap metrics will include any manual unmap IOs issued, as well as automatic unmap IOs.

External Link : Monitoring Automatic VMFS-6 UNMAP in ESXi